AI Transformation at the DOD: A Conversation with Chief Digital and AI Officer, Dr. Radha Plumb

This transcript is from a CSIS event hosted on July 15, 2024. Watch the full video here.

Gregory C. Allen: Good afternoon. I’m Gregory Allen, the director of the CSIS Wadhwani Center for AI and Advanced Technologies.

Today we have a great event about DOD’s efforts to scale the adoption and innovation of AI and other digital technologies. And we’re joined by the perfect person to talk about these issues, Dr. Radha Plumb, the chief digital and artificial intelligence officer for the Department of Defense. Dr. Plumb, thank you so much for joining me today.

Radha Plumb: Thank you so much for having me.

Mr. Allen: We’re going to cover a ton of ground about what’s going on in the DOD AI ecosystem, but before we do that, I want to talk a little bit about you and how you came to work at the intersection of national security, digital technology, and AI.

Dr. Plumb: Well, thank you, and again, thank you for having me.

You know, my background and sort of my body and history of work has largely been focused on how you get data-driven decision making into this national security space. So, my academic work had really focused on taking data applications and applied statistics and trying to use that to inform policy decisions. And then, you know, my work in the private sector was really looking at how you could take that data and help it drive both sort of different decisions and approaches within companies, and then optimizing their business solutions.

So, when I came into the department in 2021 with the deputy secretary, a big agenda item she had was really focused on modernization and driving that innovation ecosystem, and a big piece of work on that peak front was really on how you drive data – modern data practices inside the Department of Defense – so data-driven decision making, there was an early work on data decrees, as well as how you get through a digital and AI innovation into the department. And so that’s really how I came to get here.

Mirroring along with that, though, I’ll just say, is I – just before this role – was serving as the deputy secretary in Acquisition and Sustainment, and a lot of times our focus on digital solutions is on the technology – cool technology pieces, which are cool, and I’m happy to talk about them – but I think a lot of the work on the government side is really on getting those acquisition solutions to help bridge those technologies into the department. And so I think the marrying of the two digital and data background plus acquisition background has been really helpful as I’ve launched into this role.

Mr. Allen: Yeah, so you’ve got an interesting mix of academic, private sector, and national security background, and then also seeing some of the trouble spots for AI adoption in the acquisition ecosystem or in other parts of the ecosystem, so really a natural fit for your current role and we’re – great to see you here.

So you’re only the second leader of the CDAO organization, which was established only two years ago, so relatively still a very young organization. But in some sense, we’re actually pretty far into the DOD AI story. It has been almost 10 years since former deputy secretary of defense, Robert Work, introduced the Third Offset, which was a strategy that had AI and autonomous systems at its heart. That led to, among other things, Project Maven being launched in 2017, which was a flagship AI adoption effort. And so I want to get your sense in terms of the context. Where are we in the story of DOD AI and where are we headed?

Dr. Plumb: Yeah. So I think – let me cut to the end but then take a step back. I think the good news for DOD AI is we have really strong technical foundations and we have proven out now over the last year and a half an experimentation-based approach that allows us to rapidly accelerate technology solutions into fielded capabilities. But before I kind of dive in on that narrative let me just take a step back to say what is the department’s approach to AI. Our approach really comes from three pieces that I think it’s worth keeping in mind.

The first is that, fundamentally, the engine of ideas and innovation is going to be in the private sector. The pointy end of the spear on this AI technology development is coming from private companies, often with commercial or dual-use cases in mind. Second is that federal government investment is great, and I absolutely support it both in the R&D and at the procurement stage. But it really pales in comparison to the private capital investment that exists into digital technologies broadly and AI companies specifically, and I say that to say an apples-to-apples government spending comparison of the U.S. to some of our potential competitors isn’t really the right mechanism to look at that.

The third piece then is to think about, what is the government’s role here in trying to accelerate AI adoption? And I’ll, obviously, focus on the Department of Defense. And that really has three pieces. The data – we need data. AI is a data hungry process, so we need data, and it needs to be ready and usable for AI applications. There’s what I’ll call infrastructure enablement. So that’s – we call it – we’ve labeled this AI scaffolding inside the CDAO, but it really relates to a development environment with the appropriate compute labeled data and then the test and evaluation/ risk management pieces you need.

And then the third is the acquisition pathways to get the commercial technology I said before into the environment I just talked about, right. So that’s really where in CDAO we’re focusing our time and energy is how do we get those three pieces right and that’s what allows us to accelerate adoption. That’s where the strong technical foundation comes in and I’ll just stop filibustering you on this point which is, you know, the strong technical foundation that my predecessor spent the last few years building has been a really critical way for us to jumpstart into this AI environment where there’s a lot of innovation in the private sector, and now our focus is really on scaled infrastructure to enable that and the acquisition pathways needed to bring the technology into that infra.

Mr. Allen: Great. We’re definitely going to talk a lot more about infrastructure, but I also want to sort of say it seems like that’s also part of the answer for what was going to be my next question, which is CDAO is, obviously, the core of the strategy, planning, and a lot of the implementation of DOD’s AI strategy. But they’re not the only people doing important work. There’s stuff going on at each of the military services. There’s stuff going on at DIU. There’s stuff going on at DARPA. So I’d just love to hear your sense about where CDAO fits into the overall picture.

Dr. Plumb: Great, yeah. So I think maybe the way I’d start it is kind of back to those different pieces that I talked about before. On the, you know, coming up and adapting and innovating a lot of that is in the private sector, as I said. But we have some key elements that need to do that and help us think about defense specific applications or priority areas.

We have folks like DARPA and the service labs that are doing a lot of that sort of cutting-edge thinking, through especially military applications. And that’s a really important part of the development piece. Then we have a number of commercial sort of technologies that need to get prototyped and proven out in the department’s environments. DIU is really a critical accelerator there, kind of identifying best of breed in the commercial sector, matching those technology solutions to capability gaps, especially in the COCOMs. The services – the military department – really is where the bulk of our infrastructure and investment is, and they’re critical in making sure we can marry up digital solutions to our legacy capabilities and sort of onboard and scale-emerging technologies, especially in that sort of man, train, and equip space.

So where does CDAO sit? CDAO – I’ve tried to kind of bucket our activities in three areas. Enable, so that’s the policies and the acquisition strategies but also, you know, the investments and infrastructure that are needed. Scale, that is maintaining the enterprise platforms and services needed to actually have adoption. So that’s everything from our data infrastructure to, as I mentioned, that training and deployment environment we’re talking about. And then speed, and that’s really focused on making targeted investments to help accelerate capabilities to, I’ll call it, prove out the case or help solve critical gaps in a time sensitive way. We’re doing that with CJADC2, but also on priority digital solutions – things like the National Background Investigation System and key sort of AI capabilities we want to onboard and demonstrate the value of.

Mr. Allen: That’s amazing. So you’ve got the sort of three – I don’t want to call them functions, because you have this sort of official list of five functions – but you’ve got these three priority areas that you’re trying to work on. And now I’d like to hear a little bit about, you know, how that matches up to the budget. Because you’ve got, I think, in the FY-25 budget justification – for the request for FY-25, CDAO has around $650 million of RDT&E, probably some additional funding in O&M.

You could conceivably spend all of that money on any one of those three priorities that you just listed. So I’d love to hear a little bit about where you are investing, and how the money matches the outcomes that you’re trying to achieve. So if it’s all right with you, let’s start with enable, which is this policy and investments, as you just said. So what would you sort of highlight in terms of what are those investments you’re making, and what do you expect to see this year and next year come out of those?

Dr. Plumb: Yeah. So we have a couple of different investments in that enable bucket. Some of it are – some of it relates to these, sort of, foundational enablers in key areas or SUNet environment, right? These –

Mr. Allen: for folks who are not familiar with SUNet, could you just sort of explain a little bit about what that is?

Dr. Plumb: Yeah. It is a not classified – not just unclassified, not classified – development environment that allows us to take promising technologies and actually bounce it up against our data and systems in a controlled way to help us both test the capabilities in a sort of operational functions, but also do test and evaluation to include some of our responsible AI pieces. And so that – in that enable bucket additional – in addition to the environments are things like building out our responsible AI toolkit.

Which I think is a really great example of how CDAO’s a unique kind of organization. We don’t just write the policies on what responsible AI means. We translate that into tools that can then be used against specific applications to make sure they’re actually comporting with those – sort of, the technical execution is comporting with the values and principles that we want in our responsible AI work.

Similarly, we have investments in underlying cyber technology and test and evaluation infrastructure. That’s really aimed at accelerating the pathway from test – from prototype all the way to deployment, working in close partnership, for instance, with our colleagues in the Office of Test and Evaluation to make sure artifacts that come out of our ecosystem are actually useful for that test and evaluation independent review and approval.

Mr. Allen: Mmm hmm. So if I could just sort of understand how these pieces fit together. One of the things that I experienced during my time serving in Department of Defense, how – was how hard it was for private sector interesting algorithms or AI models to ever touch DOD networks or DOD data sets. And is it fair to say that SUNet is an attempt to sort of make that easier for both the government and the developer, by providing this environment where unclassified or – sorry, I guess, non-classified government data can actually find its way to be tested and evaluated with what the private sector is saying they could be brought to bear, if their systems were to be brought onto DOD networks?

Dr. Plumb: Yeah. That’s a great description. Come on back anytime. (Laughter.) We’ll put you out on marketing.

Mr. Allen: Yeah.

Dr. Plumb: No, that’s exactly right. And I think part of this, you know, kind of harkening back to my answer on, like, what does the government need to do. Like, to enable this commercial technology to actually meaningfully be adopted, we need these kinds of environments that can take the data in the department, mash it up against commercial technology, and see sort of what are the tech solutions that are actually solving our capability gaps?

Mr. Allen: And is that – is that type of service that you have, is that primarily serving the work that CDAO is doing? Or is that provided as a service to other folks in the department?

Dr. Plumb: I think largely the latter. And I guess more generally, as a principal in CDAO, while we have some analytic functions that we retain, and there’s some AI thought leadership and digital transformation focus, a big part of the maturation of the organization is making sure we have available these types of enterprise platforms and systems, because those enterprise capabilities, the scale of buying them and sustaining them makes sense to have in a central place. But the particular AI capabilities, like the digital solutions, we’re not going to be the closest to the end user. And so probably not going to be ourselves the ones sort of picking the right applications and bringing them across. We will occasionally in this speed kind of function – so in the context of, for instance, partnering with DIU on replicator, we’ll look at some specifics. But generally, this is intended to be an enterprise offering to enable enterprise-wide AI adoption.

Mr. Allen: So SUNet is part of the story that CDAO and its predecessor organization, the JAIC, and even Project Maven before that has been investing in for a while. There’s also this sort of new initiative that relates to enterprise infrastructure, perhaps, called Alpha-1. And could you sort of help us understand how those pieces fit together?

Dr. Plumb: Yeah. And they really nest together. So the way I would think about it is, SUNet is a particular environment. Within that environment now you need a bunch of stuff, right? So you need labeled data. You need T&E on that particular application. And then you need all the other kinds of documentation to get through the government, I’ll call them, bureaucratic processes. I don’t mean that in a negative way, just, like, there are a bunch of boxes you’ve got to tick to get bought at scale and fielded.

What Alpha-1 is aiming to do is for a particular use case set – in this case, sort of focused on the autonomy use cases in the department – like, let’s build out all of those pieces, the AI scaffolding as we’ve termed it, that fits together a labeled data, additional compute needed to actually test sort of more advanced AI capabilities, and those T&E resources in an end-to-end way, for a particular use case, to help us accelerate progress on that use case. But it kind of takes as a critical input into it this SUNet system – ecosystem which, as you rightly point out, has been around for a while and we know has this value proposition of being able to have the commercial sector actually come in and be able to use it.

Mr. Allen: That’s great. And so in the FY-25 budget, this entire category of thing, called foundational enablers as well as AI/ML scaffolding – that’s $150 million for AI/ML scaffolding, $198 million for foundational enablers, which includes SUNet. And these types of investments are – you know, they sound like big numbers. But the DOD is an $800 billion-plus organization.

And so I’m wondering, you know, if your goal is to scale across the department, the department is really big scale, right? Three million people, $800 billion. So is that enough money for you to provide that development infrastructure to the entire DOD? Is that enough money for you to provide labeling services to the entire DOD? Or is there some other way that you’re going to, you know, reach the scale that you need, that is separate from this pocket of money?

Dr. Plumb: So I think the way I would think about this, is this a good sort of seed money that allows it to – allows us to both flesh out what it is we’re building and begin that infrastructure process. Now, this is under the theory kind of, like, if you build it, they will come. You got to have something you can build first. This allows us to build that. If the value proposition is right – and I think we’re already seeing a lot of interest in both – from the COCOM side and the services side – then we can, you know, cost share to scale this with the military departments where the bulk of the Department of Defense investment is, as well as with the COCOMs and other partners to make sure we’re actually expanding the infrastructure in a way that makes sense.

So it’s not – would I say, like, this is all of the infrastructure investment for AI deployment in the department? No. Is this a start at showing what enterprise infrastructure could look like, and if that kind of meets the need in a – you know, again, we are starting with the use case, we’re starting with autonomy. If, in that sort of experiment-based empirical approach this looks like the right solution, then let’s build out on that. And we can figure out either within the existing structure or future budget what expansion looks like.

Mr. Allen: Got it. So right now if a program office in a military service comes to you and says, we want to do, for example, as you said, autonomy. And we’re going to need a lot of development infrastructure. We’re going to need a lot of labeling services. Right now, we’re not good at managing that stuff. Would you please manage it for us? Right now, you will foot the bill, in addition to executing the activity. And then, it sounds like what you’re saying – correct me if I’m wrong – is that, you know, once you sort of reach some kind of critical mass, then perhaps there’d be more of a cost sharing type approach?

Dr. Plumb: Exactly right. And I think maybe – to just put a slightly additional tweak on it – we have some priority use cases. If you came with a different use case, and came with dollars and wanted that to be executed, we could look at doing that in the near term as well, right? So I think there are two ways to – there are two dimensions to expand. One is more volume on the current use case, and the other is additional use cases.

Obviously, as we build out in this particular use case, we’ve got an eye towards what scaling this looks like. But we’re also taking lessons from this use case to think about what other pockets of labeled data do we want to think about bringing in and invest in, either as an enterprise way or in a cost share? And that’s going to be heavily informed by what we hear from program offices, as well as from the combatant commands, in terms of their capability needs.

Mr. Allen: So the seed money – and the way that you’ve said this is, like, building it, but building it in partnership with people who need it. That solves the traditional chicken-and-the-egg problem of DOD enterprise infrastructure.

Dr. Plumb: Yeah, that’s exactly what we’re trying to do with that – exactly right.

Mr. Allen: I get why you are going there. The next thing I want to understand is – and you have a private sector tech background so I know you’ll know the sort of weight associated with this word, but what is the flywheel for how this overall approach, you know, builds momentum in terms of what do you get back as more and more users adopt the system during that period where you’re perhaps paying for it, and then how does that build momentum overall into the flywheel-type motion?

Dr. Plumb: Yeah, so I think there are two things. One is they just hearken back to this experimentation-based approach we have, you know, in most things, which is in order to experiment, you need people to use your thing so you can experiment and actually test what works. So if we want to get an end-to-end environment that allows us to take label data, test and evaluate prototypes that come from the private sector, come out with artifacts, and then field those prototypes in a more rapid way, we need some use cases to help us prove out that value proposition. That lets us do that. And that includes things like working with – I’ll just use this T&E as an example – working with the Directorate of Operational Tests – and yeah –

Mr. Allen: Office of the Director of Tests and Evaluation – yeah.

Dr. Plumb: Yeah – to help us – like what artifact – what are the artifacts that need to come out of this environment to allow them to do their independent assessment? What do we think digital range time is that we need from the TRMC and the work that they do in kind of managing testing? What are the policy documents needed to make sure that we are comporting with the relevant policies that govern this? What are the data-sharing requirements we need to have between different elements as we pull in the data that’s needed to label and then train these models, right?

We need to work through with answers to all of those, and there is no silver bullet to that. It really is figuring it out. And to our mind, the best way to figure that out is to actually just pull a use case all the way through. You have the end result of both actually delivering a thing people need, so that’s valuable. But also you’ve learned a lot things so the next time you can do it faster and faster and faster. And I think that’s sort of our theory of change, which is let’s adopt, experiment, adopt, experiment, and the way to do that, we think, is by starting off with picking a use case where we know there is real value to our end users, the warfighters, and then just pulling that all the way through, and then seeing what we learn, and then rinse and repeat.

Mr. Allen: So, one of the things that your predecessor had spoken about, Dr. Craig Martell, was the goal of a data mesh for the entire department. Now do programs that access your enterprise infrastructure, is there some mechanism by which you are seeking to incorporate them into that data mesh?

Dr. Plumb: Sure. I would –

Mr. Allen: Making them conform to standards, for example?

Dr. Plumb: Yeah, yeah. So, I guess there are couple of different pieces to this. We have – we want to have an open federated data architecture – a data mesh is a version of that. I think that is – part of that is going to be some of the folks newly entering program offices who want to leverage label data and come back on the outside.

But a lot of that is going to be dealing with how we contract for and own our own data within the government, which is sort of going to be independent of new capability integration of digital solutions – not entirely because those digital solutions themselves will create data, which we want to make sure we own, but think of it as if you’ve got a whole bunch of sensors, and they are producing data, a program office coming in isn’t going to drive whether the government owns that sensor data and how that integration work is. So we need to have a data mesh. It’s the only scalable way to do it.

We also need to have an open architecture and sort of data marketplace that allows us to match digital solutions to the right data sets to deliver that into the department, and then we need program offices who have – program offices or, again, combatant commands like, you know, and depending on how we’re – the particular problem we are trying to solve – who can come with clear requirements, either the requirements Document Services have or capability need statements from the COCOMs, and apply those to the tech solutions to allow us to decide what tech solution is best solving the capability needs.

Once you’ve got that going, there will then – our theory – the cases, there are way more things that you will want to move through that system than there will ever be bandwidth to do it at sort of the OSD enterprise level, and so starting to build out the enterprise service where it doesn’t take us shepherding it, but there is capacity to onboard new use cases and then let the program offices do it is kind of the theory of the case for that. But we also need to fix the back-end data architecture, and that’s what the sort of Open DAGIR-driven initiative is really focused on – that sort of infrastructure or architecture piece.

Mr. Allen: We’re going to get to Open DAGIR in just one second, but I had a final question on this area. Which is: One of the metrics of success, at least in my own mind here – feel free to disagree – would be are you seeing adoption by military service program offices. So I realize some of these initiatives are in early days and some of this needs to be classified for very good reasons, but is there anything that you can share today about, you know, what type of adoption or partnerships you’re seeing with program offices for the enterprise infrastructure or for the other types of federated services that you’re looking to provide?

Dr. Plumb: Yeah. Look, I think the easiest one is, you know, going back to that Replicator example, we have program offices in each of the military departments who have autonomy systems. Those systems now need to both think about their own C2 infrastructure as well as how we execute that in an integrated way, and all of that needs to come through a sort of enterprise joint level process.

We’re working with DIU and recently just signed an MOA with them on what that process looks like. And so I think there is a number of – and just as a concrete kind of way that we are managing that we have a monthly acquisition advisory group that includes PEOs in this sort of digital space from each of the services in SOCOM to talk through some of these integration approaches – what makes sense to come into the enterprise service, what enterprise service offerings would make most sense for them versus service specific ones, and where we can lift successes from a particular service to be an enterprise offering and where we need to create new enterprise offerings to meet the different services requirements. So –

Mr. Allen: That’s great. So I’ve been thinking of DIU and Replicator in particular as, like, very much an OSD initiative. But what you’re saying is it’s already tied into program offices –

Dr. Plumb: Yeah. Yeah. Sure is.

Mr. Allen: – and military services. And so CDAO is going to play a big role in terms of the development infrastructure, test and evaluation infrastructure, for some of that work, right?

Dr. Plumb: Yeah. What I would say is, like, the digital enablers that enable these kind of –

Mr. Allen: Like data labeling or something else?

Dr. Plumb: Well, I mean, I think, you know, again, the – without getting too far into sort of classification bounds, there’s a host of digital enablers that are needed to take a number of hardware platforms, integrate them, and have them be able to meet an operational need.

DIU is rightly focused on two big pieces of that. One is what are the right commercial solutions and how do we bring them in, and they’re working with the program offices where those digital solutions are already on contract to do it. I mean – oh, sorry, those hardware solutions are on contract to do it. And the second piece is DIU is really the center in terms of coordinating efforts across the department so that the service work, other OSD work, and their work all fit together.

So in that context we’re working on this cross-cutting digital enablers work with them to help really make sure we’re able to have the right environment to test and deploy, test and field these capabilities as rapidly as DIU has in their program plans in a consistent and repeatable way.

Mr. Allen: Yeah. And previously Project Linchpin of the Army had also expressed a great deal of interest in working with CDAO. Is there anything you can share about how that’s going?

Dr. Plumb: Great. They were incredibly helpful, in fact, in harkening back to Open DAGIR in helping us think through how – what it – you know, we use these words like open and data architecture and data mesh, right, and that has to translate into actual contract language that companies can understand, and then into observable performance that the government can hold performers accountable to.

What I think the Army Project Linchpin did is not only have a good idea but actually think about how that gets pulled through into execution in a way that’s been incredibly informative, for instance, for us in CDAO in thinking about an enterprise – what enterprise infrastructure that is open really needs to look like.

Mr. Allen: That’s great. And so now let’s go right to the heart of it, the Open DAGIR initiative. So I realize this has been in the news a bit recently but let’s assume, you know, some nontrivial share of the audience has never heard of Open DAGIR, which stands for Open Data and Applications Government-owned Interoperable Repositories. So for the listener what exactly is Open DAGIR? How does it fit into the overall story you’ve been telling here?

Dr. Plumb: Let me start with how we used to do things. The way we used to buy – the way we liked to buy things traditionally in the department is in what I would call a vertically integrated way, which is I have an end effect I want – a capability. Now, in digital technologies that’s typically a user interface, right. So I want to see all of my things on one thing or I want to pick a target and be able to connect to something to prosecute that, whatever. I want to know where I can do logistics and be able to pick and apply it. I want to pull that through end to end, and so I see this user interface that allows me to make the kind of decisions at the speed I want, and I buy all of the back – I think that all of the back-end piece of that is just a price of buying that front end piece. It is not.

Mr. Allen: Yeah. (Laughs.)

Dr. Plumb: So that’s kind of – that was the problem I kind of came into which is it’s clear we need these different interfaces. It’s clear we have a bunch of mature applications that need to be scaled and it’s clear we need these different interfaces. It’s clear we have a bunch of mature applications that need to be scaled. And it’s clear we need an open, interoperable data architecture. How do we get all three of these at once? So that’s really what Open DAGIR is intended to get after. It breaks that procurement process into three big chunks.

Chunk one is the enterprise-level data infrastructure. And those you can think of in broad use case categories. So strategic C2 we did on the Palantir procurement. We just announced our intent on the enterprise analytics space, on how we’re going to compete that. So these are really buckets of a bunch of different data integrations, data management, metadata management, right, that allow you to have a bunch of data that’s connected to each other for use in particular applications. So that’s one bucket. And those are big contracts, right, because a lot of money goes into that back-end data.

The second is enterprise-level licenses. So think about this – the easiest way to think about it is, like, Word, or Excel, right? Like, you don’t want every individual who wants to use that, or every, like, 50 individuals, to buy that license themselves. We buy it at the corporate level. (Laughs.) But we were doing that for sort of – for other kinds of applications. We were doing that at the application level, right?

And so rather than doing that, for applications we know there exists sufficient demand for, let’s get enterprise licenses. And that’s a place where, just to harken back to your service question, we’ve had a lot of synergy with the services kind of understanding not just our own needs and sort of fourth estate and COCOMs – or, OSD and COCOMs, but also in the services where, again, this is a place where the government can make cost-effective investments because the leverage of the bulk buy allows you to drive down the cost per license.

Mr. Allen: And the way in which – you know, you’re presumably actually the contracting agent here. But the way in which services get their share of this application license is by putting money on the contract vehicle, or something else?

Dr. Plumb: Yeah, exactly right.

Mr. Allen: OK.

Dr. Plumb: And actually, the way we’ve done – at least, the way we’re intending to do several of these is that CDAO is really the sponsoring PSA, and we have an executive agent that’s a contracting office in a service.

Mr. Allen: Got it.

Dr. Plumb: And so it’s not unusual for these kinds of large things for then to allow sort of intra-government, intra-DOD transfers into that account on that contract. So our contract ceilings take into account the idea that you would have other – not just the dollars we’ve committed, but the broader –

Mr. Allen: But the DOD-wide customer base.

Dr. Plumb: Exactly right. And so that’s the enterprise. So, the second layer. So first layer, data infrastructure. Second layer, enterprise licenses. Third layer is – and that’s where you want sort of an agile requirements and acquisition process for this prototypes, these new ideas, digital applications to be able to come into the environment and go through an experimentation phase.

So for that, we’re using some of the more flexible procurement vehicles. Things like prototype OTs, and our Tradewinds process inside CDAO – which is a sort of expedited review process – combined with our experimentation series, called the Global Information Dominance Exercises, to sort of match capability gaps that we hear from the COCOMs in a prioritized way with technologists and potential future technology solutions. And then, an acquisition pathway, via Tradewinds, to test those solutions and see if they solve the gap.

Mr. Allen: This is interesting. So if you are an operator in a combatant command, and you have a problem set, in many times in the DOD anytime you spend any of your effort on solving this problem, you’re not really helping yourself. You’re helping your successor’s successor, right?

Dr. Plumb: Yeah, best case scenario.

Mr. Allen: Because that might not – yeah, that might not be fielded to the COCOM until multiple rotations have filled your position. But what you’re saying here is, with Tradewinds – which is this rapid acquisition pathway, and with GIDE, which is this rapid experimentation pathway, you can actually offer a COCOM operator, like, no, this is not your successor’s successor’s opportunity to benefit. Perhaps even you could benefit if you put this requirement to us and we have this rapid way to get it done.

Dr. Plumb: Exactly right. And then let’s test and experiment with it in a sort of global cross-COCOM way, so that if it can solve other COCOMs’ solutions, if we get enough demand for it, we actually have a pathway to move you from a prototype OT, from a small-scale deployment to solve specific problems, into, say, enterprise licenses, because it’s meeting an enterprise need.

So the idea is – not everything needs to, right? We can have lots of technology solutions that are particular to problem set or COCOM. And the way to buy them and build them is in this prototype rapid acquisition pathway. But you may want at least the option to, hey, we’ve solved this problem. Wouldn’t it be great if we could scale that and sustain it? And so we wanted to make sure that there is sort of moveability between these different tracks.

Mr. Allen: So you mentioned Tradewinds, which is something that CDAO runs as an acquisition pathway. But what are the actual mechanisms by which Tradewinds is faster? You mentioned OTAs, other transaction authority, which DIU also uses. But OTAs only allow you, at least in my understanding, to buy the prototype. They don’t actually allow you to use that same contract vehicle for the scaled production. Is Tradewinds – obviously they’re offering OTAs. Are there other things that Tradewinds is doing that is designed to make it faster for –

Dr. Plumb: Yeah. So I think the key to recognize Tradewinds is less on the contract pathway, though OTs are a way we do that and more on the competitive selection process, which is it’s a streamlined way to submit a video effectively about your technology solution, have it considered in a process that counts as a competitive selection process, and then if it’s selected to meet a particular contract vehicle. Then it is a competitively selected solution selected on that contract vehicle.

Mr. Allen: Wow.

Dr. Plumb: That has benefits for other things to include prototype OTs, for instance, competitively selected prototypes can be eligible for longer-term production contracts if they’ve met the key requirements at the end of that prototype phase. So it’s a – it’s intended to be a streamlined pathway in that you’ve got to prove out your technology, right? Your technology has to meet a capability need or a gap – a requirements gap, and then it needs to prove out its promise in that in a way that meets the KPPs or equivalent. But then it is a streamlined way to get onto those procurement type vehicles if that’s what makes sense.

Mr. Allen: That’s great. OK, so a combatant command has a problem. They come to CDAO. They launch this partnership. It results in actual acquisitions happening. It results in some experimentation series as part of guide, perhaps.

And now what is actually the path between that and scaled? You know, the valley of death that we’re all familiar with in the DOD environment is the man, train, and equip function that the services provide that is critical for the sort of sustainment. You know, you gave this to me this year. Who’s going to give this to me for the next 10 years? How does what you’re doing here help solve that question?

Dr. Plumb: Yeah. So let me take that in a couple of different parts. One is we always talked about the valley of death being bad. I’d just like to say up front, there are some things that should die in the valley of death, right.

Mr. Allen: Definitely true.

Dr. Plumb: So, like, my – our metric for success here can’t be everything survives forever at the Department of Defense. Point one.

Mr. Allen: Great point.

Dr. Plumb: Point two, not – I think for digital technology in particular, not all things need to be sustained. The reason I say that is our digital solutions sometimes are enduring solutions that we want to sort of continually update. I’ll come to that in a minute. But there are also sometimes digital solutions that bridge us from one set of digital capabilities to the next generation of those. So I think it’s helpful for us to think about digital technology, some of which are, I’ll call it attritable, right?

And so we want to make sure they’re funded for, let’s say, three to five years, but not, you know, enduringly into the future forever because we actually want them to be deprecated in favor of future technology and that forcing function for continual upgrades is helpful. So that’s a sometimes problem. But it is distinct from kind of a lot of the way we approach hardware and so it’s just worth keeping in mind and is very much envisioned in the software acquisition pathway. So not, like, this is not a great Radha idea, though I think it’s helpful to frame. (Laughter.)

So the third piece is then how do we get – for software solutions that need to get sustained forever how do we sustain them forever – or sustain them in a meaningful way? I think what the Tradewind pathway allows us to do is say you’ve got a competitively selected digital solution. Now you’ve got a prototype that you can demonstrate in it. You’ve got a mechanism through these enterprise licensees to actually integrate that in a directed subcontractor kind of way to however we’re managing the data infrastructure most relevant for that.

Now, I say that for two reasons. One, that allows you scale, right. So if you’re going to get scaled and sustained in that. Two, it allows you access to a procurement level vehicle, which are sort of the ID IQs or equivalents that govern the enterprise level infrastructure and enterprise licenses. And, three, with things like directed subcontracting and IP rights it allows a balance between the prime, sort of, data stack owner, the sub data application builder, and the government to allow us to negotiate important things like IP data protection, but also government ownership of sort of the intermediary products.

And that’s part of what the Open DAGIR sort of six-week sprint that the team has been working on over the last six weeks has been building out ahead of this industry day that we’re having tomorrow to sort of talk through some of those business processes in the context of our strategic C2 work.

Mr. Allen: Got it. And so, you know, thorny questions like intellectual property – what does the government own, what does the contractor own – I think CDAO wants to have the sort of here is typically the best answer, you know, and the entire Department of Defense. If you don’t want to start from scratch and figure out everything here’s an answer that we like. For most problems you can just use our answer.

Dr. Plumb: Exactly. Exactly right. And that’s where, going back to that acquisition advisory group, having that conversation with the PEOs to understand the language that they are finding effective, the language we’re finding effective, and also how we’re integrating that into our contracts and deliverables, has been a helpful construct for actually meaningfully translate open and government ownership, IP protection, these words we use, into actual language that can be enforced and also understood on both sides of the contract.

Mr. Allen: Great. So you mentioned GIDE, this experimentation series. And one of the things that GIDE has been very important in, and that CDAO is very important in, is in CJADC2. And it wasn’t that long ago that the deputy secretary of defense stated that we have actually demonstrated the minimum viable product for CJADC2 in a GIDE exercise, if I’m not mistaken. And so could you sort of explain what that means – where we are today with CJADC2, and where we’re going, and what CDAO’s role is?

Dr. Plumb: Yeah, absolutely. So let me just start with C2. (Laughter.) And I say that jokingly, but C2 is a longstanding war fighting function, right? So the question is really, like, what is CJADC2, and why do we put all those extra letters in front of C2? The answer is that to effectively execute C2 at the strategic, operational, and tactical level, commanders at each of those levels have to integrate vast amounts of data and information from all domains, all different kinds of sensors, right, land, sea, air, and then integrate it into some kind of usable interface that allows them to make decisions.

So as the volume of data coming in has increased, and the free – and the pace of – the anticipated pace at which decisions need to be made has to be faster, you see the kind of data integration and visualization problem. That’s what we’re trying to get after in CJADC2.

Mr. Allen: Yeah. And in the – for folks, you know, who had experience in the wars in Iraq and Afghanistan, the operations center – the traditional way you solve this problem is you put a bunch of people in a room. And when somebody has something that they think the whole room needs to know they, like, stand up and yell.

Dr. Plumb: Exactly right.

Mr. Allen: Which is suboptimal. (Laughs.)

Dr. Plumb: Suboptimal. And also a lot of swivel chair solutions. I, myself, having done this in Afghanistan, where you sit at one computer, you enter the numbers, and then you go look at this other computer and enter the numbers over here. And while that’s a great job for a GS-14, and gets you some deployment experience, is probably not the most effective way for us to sort of integrate data and information.

So the idea that’s in CDAO was to take an experimentation-based approach. And so pulling together a cross-department effort looking at what is the kill chain we want to close? Where are the integrations we need to solve? What are the things warfighters need to be able to integrate that information and make decisions? And let’s test on a 90-day cycle the technology solutions to come up with what is right in a scalable way.

What we did sort of in 12 months typically takes the department years. And what the deputies – and based on the parameters, assessment criteria we had, in December the deputy did indeed declare a minimum viable capability. Now, the difference between a minimum viable product and a capability is this: It didn’t just include the technology solution. And this is the most important part. It also included the operator use and sort of usability of that for executing the mission.

And that – not quite DOTMLPF because we didn’t have to change doctrine and training – but the real-time operator use and integration of it into a war fighting function is really what the deputy was looking for. She didn’t just want cool tech and cool visualization. She wanted an operator to be able to prosecute the mission thread he or she was working on all the way to execution, in order for us to count that as success. And that’s really what we focused on delivering.

Now, there are lots of kill chains and lots of improvements to be made. And so kind of we’re continuing on this experimentation pathway. But I think, you know, demonstrating, in addition to the actual technology solution, that this experimentation approach, which rather than having a sort of architecture from on high that then we build out for, you know, 12 to 24 months, and then we test for another 12 to 24 months, and then goes to a COCOM, we say: Let’s take a thing to a COCOM. Sit a software engineer and a data scientist and a war fighting operator all next to each other. And just test, and do that, and then work on fixing it, and then test it again. Like, that’s a really – a really powerful tool to accelerating delivery.

Mr. Allen: That’s amazing. So, you know, we’ve been hearing about CJADC2, or originally JADC2, for about five years now. But it sounds like you really hit your stride in the past just 12 months. So I’m curious, you know, this is currently being categorized as part of the experimentation series. And a lot of GIDE is done in partnership with NORTHCOM. You know, one of the combatant commands. When does the work that you’re doing right now start hitting other combatant commands, EUCOM, INDOPACOM, et cetera?

Dr. Plumb: Well, this last GIDE – I mean, GIDE is global. And over the last year, it’s been primarily focused on work with INDOPACOM, actually. So –

Mr. Allen: Oh, wow. So I’m out of date.

Dr. Plumb: So the priority work on GIDE – you know, GIDE as a series sits with CDAO, and we’ve really been working closely with INDOPACOM under the sort of NDS prioritization – NDS 2022 prioritization of China as a pacing challenge to prioritize the kill chains with INDOPACOM and then make sure we can have and end-to-end execution. Now that integrates key inputs from everyone, but including NORTHCOM and STRATCOM, and even EUCOM and CENTCOM. So there is a real opportunity there.

But there is also real synergy, I’ll just note, in the GIDE series of talking about the technology solutions and other COCOMS being, like, hey, I can – that could actually solve a problem I have over here and, you know, kind of lifting and shifting technology approaches to different specific capability gaps, proving out sort of what is the art of the possible with digital solutions, which I think has been a sort of ride-along benefit of the overall experimentation approach.

Mr. Allen: That’s great. And just to make sure, you know, CDAO has this big role in Replicator, CDAO has this big role in GIDE. Do those two things ever connect? Is Replicator ever at all involved in GIDE, or perhaps you – perhaps it’s more than you can share publicly.

Dr. Plumb: How can I put this? Replicator is an approach to accelerating the delivery and deployment, and they focus sort of Replicator 1 on a particular mission space of autonomy. GIDE is an experimentation series that we are using to experiment and test on how we can deliver capabilities, and we focused this last series – some of the mission threads in it – on C2. Those intersect as we need to test out the fielding of capabilities in Replicator with these digital enablers but, you know, as we look to – to hearken back to your flywheel comment – get a number of these department flywheels going. GIDE is really focused on an experimentation – a joint experimentation flywheel and Replicator on as fielding at scale, and those have to intersect to make sure the fielding at scale happens effectively, but they are both processes that need to get flywheel spinning independently.

Mr. Allen: I see. And then Replicator is one big autonomy initiative in the DOD – one of the biggest autonomy initiatives in the DOD. Another big autonomy initiative made some news recently – the Collaborative Combat Aircraft program of the Air Force. There is also the Army Robotic Combat Vehicle, other initiatives. Can you talk a little bit about how the initiatives at CDAO do or do not intersect with these programs?

Dr. Plumb: Yeah, we’re – we are working on our, you know, Alpha-1 scaffolding to be available as a environment to train and accelerate the fielding, regardless of capability.

As I said, kind of just hearkening back, whether that proves out to be the best way to provide enterprise services, I think we’ll kind of see in this same experimentation-based approach, but I do think there is a tremendous amount of promise, especially with the Army, on sort of partnering on creating environments that testing and fielding – testing acceleration and OSD-level support in sort of getting the final mile to fielding can be helpful for, and so that’s where we’re looking to try to see where is the value add from the OSD point of view, either on experimentation or on enterprise-level platforms and services, and we’re making sure we’re prioritizing our dollars in that context.

Mr. Allen: That’s great. So we’ve been talking about all the different functions and capabilities that you are providing on an enterprise scale to the rest of the department. One that we haven’t talked about yet, but one that CDAO has talked a lot about over the past couple of years, is the Advana platform. This is something that started out in the Office of the Comptroller actually –

Dr. Plumb: Yes.

Mr. Allen: – and was part of the reorganization that led to CDAO. So where are we today with Advana, and how does it fit into these other initiatives you’ve talked about?

Dr. Plumb: Yeah, so, look, Advana is a really critical part of our – what I will call our data infrastructure investments. We manage the enterprise analytic stack for the department, which allows us to integrate vast amounts of information used for financial management but also health, and personnel, and logistics, and maintenance, right, and that has a number of no-fail functions for delivery, not the least of which is things like the audit, but also, as I mentioned, personnel work and force planning, which is critical to readiness.

Where we are in the journey is we are just entering into the period of time where we will need to recompete the overall contract for Advana, and we’re working to recompete that in the construct of Open DAGIR, which again means that we need to compete the data infrastructure, the enterprise applications, which will probably be more than one, you know, particular application set. And then this, you know, prototype pathway for analytic applications that, you know, want and need to be tested and evaluated on that Advana stack.

I think the benefit here is we can really create clear data infrastructure investment on this really critical back-end data, those data pieces, in a more modular way than we had before. Advana is like a victim of its own success. It scaled tremendously over the last few years, and that data infrastructure itself, we’re doing some internal upgrades, I think will benefit from an overall look at the data engineering and kind of what the right solutions are for the back-end architecture. And so looking forward to going through that competitive process to see kind of who’s interested in working with us on that.

Mr. Allen: That’s a remarkable statement you just made. Advana is, in some ways, a victim of its own success. What do you – what do you mean by that?

Dr. Plumb: I just mean, you know, it’s the story – a story as old as time. (Laughter.) And it’s super common on the commercial side too, right? Where you – basically, you have data infrastructure you make, and then if you do it right your demand for that data infrastructure outstrips your initial build. And then you’re going to upgrade your infrastructure and then build on your application until you do. And that’s kind of the sawtooth pattern of digital deployment.

Mr. Allen: But instead of customers paying you at the exact same rate at which your user growth is, you’re going to the DOD budget request process, asking to add capacity to Advana because your user demand growth has exploded.

Dr. Plumb: Yeah. And I think this is – we have had – we’ve had both, right? Customers both paying for it and enterprise-level investment in it. And now we’re at the maturation place where we actually have a pretty good idea of – a very good idea of what the user base is and needs for right now, and what the applications that need to be built on it are, at the enterprise level. And so it’s, you know, a good time for us to just take stock and look at all it has delivered, which is truly amazing.

If you, again, to your point, think about it as starting in the comptroller space on financial management and growing in just two years, you know, a hundredfold in terms of users and across all domains of enterprise analytics inside the department, to be the system of record for that, right, that’s truly tremendous. And so looking now at what is the next kind of journey for Advana look like, and how can we make sure we built the right back-end data architecture, the right enterprise-level analytics, and we have the right acquisition sort of strategy to make sure that can launch into the future just as successfully as it has over the last several years.

Mr. Allen: Amazing. So CDAO has so many different jobs, right? You’re building enterprise infrastructure. You’re developing test and evaluation technologies. You’re writing policy and strategy designed to make everybody else’s life easier in the Department of Defense. You’re leading the way on CJADC2. You’re leading the way on a lot of different parts of autonomy. And it strikes me that it’s almost as though you’re being asked like, oh, Dr. Plumb, would you please just field an Olympic-quality basketball team, and also an Olympic-quality chess team, and also an Olympic-quality, you know, swimming team? You’re being asked to be good at a lot of different types of things. And I’m curious, you know, how you think about it in your own mind, about how do you manage this diversity of focus areas so that the whole is more than the sum of the parts? And how does that strike you?

Dr. Plumb: Yeah. Let me start with, I think that’s the – that was the value proposition of CDAO when it was created, right? We had these real pockets of excellence – the Chief Data Officer Office, which is doing great work on data integration. We had the comptroller team doing great work on Advana. We had JAIC really at the forefront of AI development. You know, we had Defense Digital Service doing sort of our digital SWAT team and response.

And the idea of putting it together was that the sum was going to be greater than each of the individual parts, in part because there are lessons to be learned from the different parts for each other. In part, because that enablement function – the ability to have policies and governance that tie these sort of different elements of the digital transformation together – can really critically integrate and deliver things that couldn’t happen with individual pockets.

And I think because the CDAO itself has kind of proven out this – what we’ve laid out as this hierarchy of needs, which is you have a number of data-hungry processes in the department. Be it AI applications, or analytics, or the need for data-driven digital solutions, right? And so the driving progress on quality data, and then having this meaningful analytics, and then having responsible AI adoption as a sort of nested function within us, has allowed us to scale in a responsible and meaningful way for the department, that I think is tremendous.

The last thing I’ll just say is we are incredibly benefitted by inside CDAO having an extremely talented workforce. These are people who definitely are not there for the government dollars, and are bringing their considerable engineering, software, policy, analytic talents to meet the mission for our warfighters and for the department leadership in a way that really just tells you kind of the value of their sort of patriotic commitment to doing this for the warfighter. And I think the way I think about management for that community is making sure I am putting in the time and effort to enable and make sure they are getting the things they need to do their jobs and try to stay out of the way where I can do things that make it harder for them to do their job.

And I think, you know, a lot of times there is an underappreciation for the civilian workforce and the talent they bring to solving problems for the warfighter, and I think CDAO is a great example of extremely talented civilians and their military counterparts that are working on critical functions and delivering, as you said, a whole host of priority initiatives day in and day out.

Mr. Allen: Great. Now I want to just push you a little bit on this point. There is a quote from Steve Jobs that I like – the former CEO of Apple – where he says, “Focus is about saying no.” So you’ve got this quite broad portfolio of things that you are working on. Is there some rules in your own mind that sort of say, what might be good ideas for the department, might be good ideas for even part of OSD, but CDAO is not going to go there. They’re not going to work on that because we’re focused on these other – you know, a pretty broad portfolio of actions that you talked about?

Dr. Plumb: Yes. I totally agree with that. I think about this in two ways. First, sort of at the strategic level, my general thinking is we should ask sort of what are we doing – like, you know, what’s the value proposition of this thing. And then why does it have to happen in OSD at the CDAO, right? We need to be able to answer that question for everything we are doing because our budget is our time, and our time is finite.

Second, when I first came in, in the first 30 days, I tasked each of my direct-report directors to do what we call the Three Ps exercise, which was to prioritize the set of things we absolutely must get done; to pace – i.e., slow down – things that we want to do but maybe don’t need to try to execute as fast; and the pause things that were not essential for execution. And the idea of the three Ps exercise is to allow you to focus your resources on the things that actually must get executed – your no-fail missions – and then allow you to pace, to slow down, to have sort of the additional area where things you want to make progress on.

But you know you can pull resources from it if you need to for prioritized, and pause – and pause – very importantly, pause; not stop, not kill, not end, right, but pause things you don’t need to do right now. That pause bucket, I’ll just note, can have lots of ideas that we may want to do later or that aren’t right for CDAO that we can pass to other partners to execute, but forcing the discipline sort of by component within CDAO to do that is part of how – exactly right, like, we’ve got a bunch of missions we absolutely must meet, and so we need to align our resources and execution to that. And I think we’re doing our job to do that.

Mr. Allen: That’s great. And in your case, you are working with AI, which is the technology that is not standing still so you decide what you want to focus on, and then something like the large language model revolution comes along. And this is an area where CDAO also plays a big role. You’re leading Task Force Lima on the part of the department. Can you share a little bit of an update about sort of where you are in terms of thinking about generative AI and what CDAO’s role will be in the generative AI story as Task Force Lima presumably winds down at some point.

Dr. Plumb: Yeah, so kind of where we are, candidly, in this story – Task Force Lima is providing its recommendations up to me, and we will provide that to the deputy, both to prioritize the use cases along which we want to begin our experimentation journey on AI adoption, but also to help us understand what the guidelines and guardrails we need to put in place as we begin testing and using generative AI.

I think, to my mind, the CDAO’s role is, you know, twofold in this space. One is, really on that guidelines and guardrails. Like how do we give you the right left and right limits to allow experimentation in a federated way inside the DOD ecosystem because that’s really what’s going to let us better understand and realize the promise of generative AI. It allows what I’ll call responsible risk taking; like, we want people to take some risks, we want you to be uncomfortable, some of these things should fail. We don’t want you to take risks where you shouldn’t be taking them and let us fill that out for you. So that’s one bucket.

And then the critical enablers. And, look, generative AI is an interesting case where, you know, if you think about traditional infrastructure investments, you might want, like, three-quarters – two-thirds, three-quarters of your infrastructure investments, and then, you know, the remaining one-third to one-quarter being sort of your front end user interface and user testing.

These generative AI models are hugely infrastructure dependent. They depend on a tremendous amount of compute, which has energy requirements. They depend on, for DOD, thinking about, like, cloud versus on-premises compute capabilities, and what their transport looks like. So we’re also now taking a look with our colleagues in CIO at just what does it mean to make the department ready for this? And what does that have to look like in future budget requests? And so I think those kind of are the two big areas for us to look at while we enable the enterprise to look at pilots and experimentation on how Gen AI can help them.

Mr. Allen: Fabulous. So there’s one final thing that I just want to ask you, because I’m sure there’s a lot of people who are wondering this. The ADA Initiative was something that CDAO announced early in its life. Part of the story was related to helping out EUCOM right after Russia had invaded Ukraine. So I’m wondering if you could provide an update on ADA, and what has happened, and where we are today with that initiative.

Dr. Plumb: Yeah. So those teams are, first of all, just an amazing set of individuals who are working closely. These are AI and data leads who work closely with the combatant command to help integrate data at the COCOM, forward deployed, and then build out some of the initial applications – or, help them identify and build out the application. A core part of what they also do is help identify the digital readiness of a particular combatant command, provide that report to leadership, and then help leadership sort of understand where changes need to be made, or not.

Where we are in the journey is the ADA teams are out, they’re deployed. We extended them for one year, and now we’re working through the process to go up to DOD leadership on what the long-term sustainment plan for them is. It’s clear that they are extremely valuable to the combat and commands –

Mr. Allen: This is a talent pool they wouldn’t normally have access to, right?

Dr. Plumb: Exactly. Not a talent pool they normally have. I think having – there are some questions about how we want to central – how much central management you do or don’t want. There’s both value in this sort of decentralized, very forward-deployed COCOM ownership, but also having communities of practice, and best practices, and information flow back and forth to the COCOM. So we’re really looking at what are models to kind of take the lessons we’ve had from the last couple of years of ADA and really make the program institutionalized, but in a way that builds on the successes and helps mitigate future risks.

Mr. Allen: And I realize a lot of this is classified, so it’s fine if the answer is you can’t tell me anything, but is there anything about the war in Ukraine in particular that you’re able to say here? I remember, you know, ADA was part of this conversation a while ago.

Dr. Plumb: Without getting into too many specifics, I think what has been critical about that ADA team is its ability to, at the sort of end point, help solve digital problems with everything from tracking flows of things coming in and out, to tracking dollar, to building visual interfaces, right? So there’s just a number of different digital solutions that you can see are – in a world where things are moving quickly, those digital solutions can be built quickly and have real impact. And that really helps us see the value proposition of having this digital talent pool deployed with our war fighters with our combatant commands, to make sure they’re getting those solutions in real time as they need it.

Mr. Allen: Well, Dr. Radha Plumb, thank you so much for coming to CSIS today. It’s really phenomenal to hear about your different experiences across the private sector and across the different parts of the Department of Defense have really enabled you to look at this problem and go out and tackle it, handling so many different – I guess I should say – juggling so many different balls simultaneously. But I really appreciate you spending your time here with CSIS and helping us understand this.

Dr. Plumb: Thank you for having me.

Mr. Allen: This concludes our event. You can watch the livestream available at CSIS.org, and it will be available almost immediately after the conclusion of the event for rewatching or sharing. Thank you again.

(END.)