Scaling AI-enabled Capabilities at the DOD: Government and Industry Perspectives

This transcript is from a CSIS event hosted on March 26, 2024. Watch the full video here.

Gregory C. Allen: Good morning. I’m Gregory Allen, the director of the Wadhwani Center for AI and Advanced Technologies here at the Center for Strategic and International Studies, CSIS. Today we have an event covering ongoing efforts to scale AI adoption in the Department of Defense and we’re joined by two of the leaders in this effort from the office of the Chief Digital and Artificial Intelligence officer, the CDAO organization.

To my right is Navy Captain Xavier Lugo, who is a division chief in the algorithmic warfare group at – for artificial intelligence and scaffolding and integration at CDAO. And he’s also the commander of Task Force Lima, which is the DOD’s task force related to generative AI adoption.

And then to my left we have Air Force Colonel Matthew Strohmeyer, who is the combined joint all-domain command and control experimentation division chief at the CDAO.

Gentlemen, thank you for joining me today.

Colonel Matthew D. Strohmeyer: Thanks, Greg.

Mr. Allen: Just before we get into the actual meat of the panel I’d like to know a little bit more about you two and how you got into military AI.

So Captain Lugo, could we start with you?

Captain M. Xavier Lugo: Sure. So I’ve been in the Navy for 30 years, close to, this March – coming March and I’m a supply officer – a logistics officer so basically been dealing with data throughout my whole career.

I’m a mechanical engineer by degree so been always interested in the fusion of those two, right, of engineering and data. As I’ve been progressing through my career, I finally ended up right before this tour on modernizing the logistic IT systems for the Navy within OPNAV, taking over 200 legacy systems and combining them into brand-new technologies and capability-based technologies which include, of course, data centricity, artificial intelligence, analytics.

From there, I was given the opportunity to join what was then the JAIC.

Mr. Allen: The Joint Artificial Intelligence Center.

Capt. Lugo: Yes. And then for – I was a JAIC-ster for about 30 days and then we converted into the CDAO, which now, obviously, I’m in this field. I’m also an ops research analyst. So it all merged. That combined with the experience of data and analytics.

Mr. Allen: Great.

Capt. Lugo: So I could go into more detail but that’s a good summary.

Mr. Allen: This is a common phenomenon. The operations research – the ORSA community is a lot of the data crunching community within the military services and there’s a lot of overlap between that community and the AI and machine learning community.

Capt. Lugo: Absolutely. Yes.

Mr. Allen: Great.

Captain Strohmeyer, you.

Col. Strohmeyer: Greg, good morning. Thank you so much for the opportunity to be here and thank you to CSIS for this platform to allow OSD and CDAO really to kind of get our message out and let the public know about what we’re doing.

So I’m a 23-year Air Force active duty colonel. I started my time as a fighter pilot in the Air Force but then transitioned over into the military strategy and kind of operational war plan planning world and doing that out in the Indo-Pacific and other places across the globe, and eventually landed at NORAD and U.S. NORTHCOM and there was directed by the commander to try to accelerate the modernization of the homeland defense capabilities that NORAD and NORTHCOM were focusing on but to use a new approach – instead of to use kind of the traditional requirements approach, still a very important part of what we do, but to use – because a lot of our homeland defense capabilities needed to be modernized in terms of their data, in terms of how we got sensor data, how we got data about where our operational forces were to be able to defend the homeland, we use an experimentation approach to be able to do that.

And so I started leading a series of experiments when I was at NORAD and NORTHCOM that became known eventually as the GIDE, or Global Information Dominance Experiments, and the approach then was very different in the sense that rather than trying to, you know, go about every 12 to 18 months and do an exercise, which we normally do, instead to take this rapid iterative failure-seeking approach to try to field capabilities rapidly.

And so in this – in the context of starting GIDE and building up that capability we really started, one, to gather the data that we needed for homeland defense but then we also started working with AI-based capabilities to allow us to use that data better to gain more insights on how we can defend the homeland better, how we can better understand what our adversaries are doing, sort of partnering with organizations like Project Maven originally at USD(I&S), now at NGA, and then that eventually morphed into GIDE being brought up to the deputy secretary of defense level and now being run up at CDAO for Secretary Hicks.

And so that’s what I do now is I run that GIDE series of experiments up at OSD.

Mr. Allen: That’s great. And let’s keep talking about GIDE because that’s one of the more interesting things going on anywhere in the DOD right now in terms of accelerating AI adoption.

So you changed jobs but kept working on the GIDE series.

Col. Strohmeyer: That’s right.

Mr. Allen: And can you just help us understand what problem exactly was GIDE created to solve and what is the mechanism by which it seeks to solve that problem?

Col. Strohmeyer: Yeah. Really good question.

So GIDE really started as an accelerant to be able to move at least NORAD and NORTHCOM originally but now really to move the entire Department of Defense to modernize a lot of our capabilities when it comes to data, when it comes to the way that we make decisions.

So, broadly, I would say that GIDE was designed to help us realize decision advantage for the U.S. Department of Defense against any competitors or adversaries.

What do I mean by that? I mean, the way that we, whether it’s at the strategic level if we’re trying to decide should we move this carrier strike group here or should we move these munitions here or how can we create a deterrent effect against our adversaries at the strategic level to be able to get to that decision in a better, faster, more data-informed way and then – but below that at the I would call the operational level, the way that we make, for example, fires decisions.

That’s how we would, for example, complete a kill chain against an adversary, allowing us to be able to do that as well – that work of whether it’s a maritime strike munition going into a maritime target, or whether it’s a land attack munition going into a land territory, or a space or a cyber effect, being able to close those kill chains better and faster than we’ve been able to do in the past and, importantly, in a more joint way than what we’ve been able to do in the past.

The services do a great job of being able to close their kill chains. Our job is to try to bring that together in a more joint way so the services have their data in common and can make joint decisions together.

And so that’s essentially the thing that GIDE was created to be able to do, first at a combatant command level and now at a Department of Defense level. And the other thing I’d say related to it is the approach that it takes is different in the sense that a normal way that the Department of Defense would ready our forces or test our capability would be via an exercise series.

The exercises are very important but they are on a 12- to 18-month timeline and so the learning that you accomplish in an exercise has a really long feedback loop. Important feedback loop, but it’s long.

Under the GIDE series under the direction of Secretary Hicks she has us conducting a GIDE experiment every 90 days and so we essentially go from lessons observed, lessons learned, and lessons applied not only within a 24-hour cycle during a GIDE but then every 90 days outside of those GIDEs.

So it’s a new way for the department to be able to rapidly learn lessons and turn those lessons into fielded capabilities.

Mr. Allen: And when most military service members are changing jobs every two years, right, that’s the difference between going through one exercise in your current term –

Col. Strohmeyer: Absolutely

Mr. Allen: – and eight, you know, GIDE series.

Col. Strohmeyer: Absolutely. Yes.

Mr. Allen: So recently Deputy Secretary of Defense Kathleen Hicks stated that the minimum viable capability for CJADC2 is real and ready now, and I know this is in your title. It’s a critical part of what GIDE is trying to accomplish.

So what exactly did Deputy Secretary Hicks mean when she said it’s real and viable now and what role did GIDE have in making that happen?

Col. Strohmeyer: Yeah. Great question, Greg.

If I can unpack first what is CJADC2. For a lot of the listeners –

Mr. Allen: Mmm hmm. Sure.

Col. Strohmeyer: – they might say that’s a very jargony military term – what does that mean?

So it stands for Combined Joint All-Domain Command and Control. Essentially, it is warfighting for the 21st century as opposed to in the past where we had to have each individual service conducting individual service decisions about how we move forces, how we employ those forces.

CJADC2 is a way of – it’s not a single thing we buy but it’s a way of warfighting where we connect the entire joint force together, connect the data across the joint force, connect the decision-making systems across the joint force so that we can make better, faster, more risk- and data-informed decisions, essentially.

So in the context of that and in context of CJADC2 and this capability that the department has been seeking after fielding we – the deputy secretary about a month ago in a very historic announcement at the CDAO symposium said that we had fielded a minimum viable capability for CJADC2 via the GIDE series last year.

So how do we go from experiments to actual fielding of some level of capabilities? Well, the GIDE series – last year we did GIDE V, VI, VII, and VIII. It was the first number of the GIDE series that were done not just at a combatant command level but at the Department of Defense level both with the combatant commands with many of our allies and with the Joint Staff also involved.

And so we went through a process of learning from GIDE V through GIDE VIII that started with GIDE V asking a question can we conduct joint force experimentation to field things at scale across the department. We learned yes, we can do that.

And then GIDE VI and VII really looked to refine the thing that we were trying to field or this heterogeneous mix of things that we were trying to field to be able to get a(n) actual JADC2 capability out into the field.

And then GIDE VIII was really our delivery event where we said did we actually – were we able to successfully deliver something that warfighter said yes, this is helping me – this is helping the things I need to do every day, and we fielded that across two mission threads that I can go into later if you’re interested.

Mr. Allen: That’s great. And one final question on this topic. You know, when you say that there’s a delivery event in GIDE VIII, you know, we’re talking about the valley of death in the Department of Defense, which is this legendary problem set whereby sometimes in exercises, sometimes in research and development environments, really exciting capabilities are demonstrated and as you said, warfighters say, I want that.

But how does that get translated into the actual bureaucratic machinery of the Pentagon for, you know, man, train and equip and sustain? Is the GIDE capability delivery that you’ve achieved is that now translated into permanent programs of record or permanent sustainment activities or permanent COCOM acquisition activities? Or what’s the story there?

Col. Strohmeyer: Yeah. Another really, really good question.

So, one, we think that the way that CJADC2 will continually and iteratively be fielded will look different than a normal program. This is not hardware. This is mainly software and data connected to hardware and connected to warfighters, and the way that we are going about it is different in the process that we’re using.

One, we recognize that whereas you might have a piece of hardware that you research, you develop the requirements for, you acquire it, you send it out to the force with all the DOTMLPF, the training that would need to go with it.

With JADC2 it’s different. Rather than having one organization that researches, develops the requirements, acquires it, and then gives it to all the users we want to have a much more warfighter involved process with it in the sense that we want to have – and this is what we did last year during the GIDE series – we have software engineers sitting directly by warfighters at combatant commands, at subordinate echelons below them, at the Joint Staff, and when they go through a GIDE experiment the warfighter is giving direct feedback literally next to them to the engineer and saying, this is what works, this is what doesn’t work, this is really what I need to accomplish.

So both they can get that feedback but then the engineer can actually live and see this is the result of the work – the coding work that I did. This is how it’s impacting that warfighter. That’s kind of core to what we’re doing.

So that iterative process is very central to what we’re doing to be able to field the capability. But CDAO is also uniquely given the responsibility of being a principal staff agency responsible, right, to the deputy secretary of defense and also having a lot of those capabilities inherent in us to be able to acquire those things, to be able to have organizations that can start to look a little bit like a program office to be able to make this a reality. And I was very glad to get the appropriations bill passed through this weekend so that we can now start moving out at scale to allow CDAO to be able to do that.

Mr. Allen: A tremendous amount there that I could continue talking all day.

But I want to bring in Captain Lugo here because you’re leading not just one but two of the most exciting initiatives, and whereas Colonel Strohmeyer is closer to the combatant command and the adoption side of the equation you have two efforts that are a little bit closer to the development and the creation side of the equation and that is Alpha 1, which is intimately related to the AI scaffolding initiative, and then Task Force Lima, which is related to the department’s adoption of generative AI.

I want to start with Alpha 1. This is now in the FY ’25 budget request. There’s some public language around what the DOD is trying to accomplish with Alpha 1 but I think most people are probably hearing about it for the first time. So what exactly is this initiative and what’s it trying to accomplish?

Capt. Lugo: So it was great that you started with him because he is showing exactly part of the reason why Alpha 1 exists, right? So, first of all, as Nomad mentioned, we – the CDAO has some unique missions that are not typical to other PSAs that overlap with some traditional program office type efforts, right?

So as GIDE goes on one of the things that they noticed is, you know, there’s some sustainment that has to go on with those initiatives, right, and there’s certain gaps in the enterprise that have to be filled especially in the AI/ML world, right, which is really where the algorithmic warfare and subsequently where Alpha 1 resides.

So what Alpha 1 is, is it’s really a portfolio. It’s not a platform. It’s a portfolio of capabilities and services that will enable AI/ML scaffolding at scale on the enterprise. So as mentioned, it’s been resourced for the first time this year. So it’s – but we’ve been doing all the planning efforts so that –

Mr. Allen: When you say resourced for the first time are you referring to the FY ’25 request or the FY ’24 this bill that was just passed?

Capt. Lugo: The bill that just got passed, right? So and we have – our plans are basically to go through capabilities that, again, are required at the enterprise level and what I mean with that is we’re not building the whole AI/ML scaffolding loop at once and then hope for people to join it.

What we’re doing is we’re finding what those most demanded areas are whether it is by the services, by program offices, by experimentation that chose these requirements and fielding those in a priority order as needed.

So that does two things. One of them is it targets the most important needs at that moment and it also gets enough time for capabilities to join when they’re most refreshed in the industrial world. So –

Mr. Allen: So who is –

Capt. Lugo: Yeah.

Mr. Allen: Just in the model that you just described you’re talking about providing the services the sort of portfolio of services that folks who are trying to develop AI capabilities or autonomy capabilities want. But who is your customer in that sense within the DOD environment?

Capt. Lugo: Yeah. So let me – I’ll answer that two ways. One of them is the customer is really not so much the COCOMs yet but mostly services and specific program offices.

Mr. Allen: Interesting.

Capt. Lugo: OK. But let me go via example, right? So the first capability that we have made available is data labeling. So data labeling was found to be a huge gap across the department and in the sense of program offices that were very mature, already had their own data labeling services, but that came with constraints and caveats.

So, for example, the data was not necessarily owned or wasn’t necessarily shared or ability to share that data. That’s one of them. Other pieces are that the tools were not really standardized or interoperable.

So by doing a centralized – centralized is a loaded term. By doing a – at scale –

Mr. Allen: It’s centralized, but it’s not a compulsory centralization.

Capt. Lugo: Right. Right.

Mr. Allen: It’s a(n) optional centralization –

Capt. Lugo: Well, it’s centralized from the perspective of the service is available centrally, but it’s not centralized as a push to anybody to use.

Mr. Allen: Yeah. There’s no “thou shalt” in this story. Right. Yeah.

Capt. Lugo: Correct. So but by doing this – for example, three programs that are transitioning right now which are Smart Sensor, Maritime Object Detection, and Harbinger – Smart Sensor transitioned to the Marine Corps, MOD is a Coast Guard program, and Harbinger is a Navy program – those three are now part of the data labeling service and we provide data labeling as a service to those three programs.

They don’t have to worry about it. They don’t – so now the program office really what needs to be worried about is building models and integrating those models into the weapon systems.

Another example was those were programs that were part of the JAIC actually in the beginning and now they’re transitioning to the services. But the Army came in with Army Project Linchpin and they said, we like that model. We want to be part of that model. And basically we accelerated them from their milestone schedule from 18 months to two years into starting with the capability of data labeling just because it’s available now as a service to the DOD.

Mr. Allen: Just to make sure I understand you correctly here, you know, data labeling is critical to anytime you’re doing most categories in machine learning, especially supervised machine learning.

Capt. Lugo: Correct.

Mr. Allen: It’s foundational to doing that. And what you’re saying is the previous state was the DOD had all these programs of record. They were all instructed or excited to do AI and they all ran into the same challenge of data labeling, and they went and solved it their own way for their own sort of immediate needs and there was this missing economies of scale and standardization because they didn’t necessarily have the expertise to write the contract, they didn’t necessarily have the expertise to run the data labeling effort correctly, and you now provide that as a service.

Capt. Lugo: Not only that, I can level load the capacity, right? So if a program thought they needed a certain amount of labeling services and now they need more or they need less I can level load that as an economy of scale versus either wasting or stoppage at a certain program. So that’s just one example.

Mr. Allen: You allocate across this, you know, portfolio of customers. Yeah.

Capt. Lugo: We can allocate as necessary, right? And I’m not limited as to how many I can onboard, right? So, again, there’s a lot of agility there. So now as we’re going into other lines of effort like, for example, autonomy now we’re able to onboard those into this same labeling as a service for – I don’t want to call it contract service. And then by doing that they have a(n) instant on that they can go in and join.

So that’s just one example of the whole AI/ML DevSecOps. We got other pieces with containerization. We got other pieces also with the actual instrumenting of models, the model repositories, also data management platforms.

All of those services are going to be interoperable – that’s the key here – and available so that program office has much as they need or as little as they need. They can go in and just join into the – into Alpha 1.

Mr. Allen: So is it fair to say that, you know, your mission, loosely defined, is to make life easier for program offices who want to do AI or add AI?

Capt. Lugo: Correct, and easier in the sense of agility, also easier in the sense of resourcing and lessons learned. So and also best of breeds, right? So, and again, we’re not prescribing any particular technology.

We’re not prescribing any particular standard. All we ask is that your data is available for everybody and that you’re interoperable so don’t come up with a customized way. Then it won’t fit into the Alpha 1 program.

Mr. Allen: So some terms that we’ve heard from the CDAO including at the recent CDA symposium are things like AI scaffolding and the data mesh, and could you just sort of explain, you know, number one, those terms? What do they mean?

You’ve already explained a little bit about AI scaffolding in particular. And then what is the overlap with Alpha 1? Is Alpha 1 a synonym for what the CDAO is doing in AI scaffolding or is it one part of it?

Capt. Lugo: So let me first start with data mesh. First of all, neither one – we both play in the data mesh but we don’t own the data mesh. The data mesh is part of the policy arm of CDAO.

However, what it is, is really it’s all about the interoperability and the transparency of the data. That’s really what it is, and the transport of that data across. So, obviously, GIDE and JADC2 is very much, I would say, the reason you need a data mesh.

Alpha 1 or AI/ML scaffolding is leveraging the data mesh in order for us to make the data available across all these program offices without having to centralize the data necessarily but be able to reach the data where it’s at in authoritative sources.

So that’s my simplistic way of describing the mesh.

Mr. Allen: Well, can I just – I want to sort of understand the strategic logic underpinning the data mesh and here’s what I think I’ve heard you say.

So CDAO is going to be providing some resources to organizations that want to do things like data labeling through Alpha 1 but a condition of accessing that resources is that you have to make your data under a standardized format and you have to make it available to the broader DOD. Is that right?

Capt. Lungo: So I got to be careful with the word standardized. This is not about standardized formats. It’s about being able to have your – you got to have ontologies that are accurate so we can actually –

Mr. Allen: So it’s not standardization so much as clarity and transparency?

Capt. Lugo: Correct. Yes.

Mr. Allen: Got it. So they have to make their datasets transparent and accessible to the broader DOD and then that’s going to start sort of its own rock rolling down a hill in terms of gathering momentum because as more and more programs use Alpha 1 then more and more of the attractive portion of Alpha 1 is going to come from accessing this data mesh versus accessing money from CDAO. Is that fair to say?

Capt. Lugo: Yes, that’s fair to say and you can expand data now to models and other pieces of services, right, and other tools, right? So –

Mr. Allen: Right. So you’re starting with data labeling –

Capt. Lugo: Correct.

Mr. Allen: – but that’s only the first set of tools that folks are going to get when they partner with Alpha 1?

Capt. Lugo: That’s Correct.

Mr. Allen: And it seems like, you know, because those three efforts that you mentioned – Harbinger, Smart Sensor, and Maritime Object Detection – obviously, they’re in partnership with other parts of the DOD but they sort of have their origins in those partnerships with CDAO.

Specifically, Linchpin is a major Army program of record and this partnership seems really important to CDAO and Alpha 1 is kind of at the heart of that partnership. Could you sort of elaborate a bit on the state there?

Capt. Lugo: Correct. So their timing was perfect, right? So as they’re briefing Linchpin, Alpha 1 is also being kind of designed. And so they saw the opportunity, we saw the opportunity, to partner and basically, well, we’re going to be providing this – these as a service. Why don’t you take advantage of it and then also then – they’re a PEO so they have multiple programs under that.

So now they’re going to be providing basically data labeling as a service through Alpha 1 to their Army program offices and what that did was just because we were a little bit ahead of them, not necessarily any other reason, right, programmatically – and I think Nomad talked about that – the traditional POM cycle and the traditional way of planning – we had the advantage of providing this quicker to them.

So now instead of them having to wait for a particular milestone to start providing that service it’s probably the – there’s going to be label data. There is already. Labeled data is going to be available at the point where the program office that has got the weapon system ready to receive that data versus being more serial.

So that is a –

Mr. Allen: So this is a path for you to over the longer term get plugged into a lot of program offices across the Army, you know, starting with the Linchpin Program executive office but really –

Capt. Lugo: Correct.

Mr. Allen: – Army is getting direction to utilize Linchpin where possible for AI, inject it into programs of record or creating the programs of record, and now CDAO is going to be a part of that journey.

Capt. Lugo: Absolutely. And we can take that model and expand it to others.

Now, still, like, with everything Alpha 1 really what it is, is about getting the AI/ML scaffolding across – you asked me and I didn’t answer yet what is the difference, right? So the AI/ML scaffolding really is about – the term scaffolding is appropriate because it’s about having the proper safeguards, the proper guardrails, the proper systems for AI/ML to actually flourish and be built.

We are not building models. Again, let the person that owns the problem own the solution as well, right?

Mr. Allen: The program of record and industry partners will build the model.

Capt. Lugo: Or the COCOM or whoever needs – whoever needs that solution, again, just like Nomad described, you got that soldier or sailor or Marine or airman right there coding or telling the coder what to do, right, because they own the problem set. They’re the best equipped to solve that problem.

But what they need is the tools. What they need is the scaffolding to make sure that those tools meet all our other requirements, right? We haven’t talked about the bad side of AI, right – the actual, you know, risks that may be out there. When you have the right scaffolding we take care of at least assisting and so that those risks are mitigated.

Mr. Allen: Mitigated. Yeah.

Capt. Lugo: Correct. And then you get agility. Then you get the speed that you need to solve those problems. So that’s the intent. So are they analogous? No. Alpha 1, again, is that portfolio to –

Mr. Allen: And Alpha 1 is sort of the tip of the spear of the larger AI scaffolding.

Capt. Lugo: Correct. The AI/ML scaffolding is larger than Alpha 1.

Mr. Allen: Got it.

Capt. Lugo: But Alpha 1 is the one that is starting to get momentum to enable – to start enabling AI/ML scaffolding. We do have platforms out there that are already part of AI/ML scaffolding and they have their own way of mitigating those risks and ensuring capabilities are produced.

Mr. Allen: So, Colonel Strohmeyer, we just heard, you know, that GIDE is also a part of the data mesh story and also a part of the AI scaffolding story and potentially even, you know, poised to benefit from Alpha 1 sooner or later. Could you just elaborate a little bit about how GIDE fits into those ideas?

Col. Strohmeyer: Yeah. Great question. So I’ll answer it first in terms of the relationship with Task Force Lima Alpha 1. GIDE really is an experimentation venue with warfighters for those capabilities.

So as they work to build out the enterprise capability for AI and allowing new AI capabilities coming in from services GIDE provides a venue to then experiment with a warfighting mission on one of those capabilities.

So a good example of this is we just finished up GIDE IX last week and during it we did a single blind test with multiple combatant commands of an AI capability to be able to take logistics warfighting workflows that we have to go through – for example, analyzing what a good vehicle or means might be to be able to get some sustainment capability from one location to another location – and the participants were required to be able to generate in a very short period of time what is a normal thing for a J-4, so an actual logistics leader, to generate, what is their recommended path that they would take.

And then some of them had access to a generative AI tool that allowed them to do that but it allowed them to do it in a way that they could see where it was generating information to make sure it wasn’t hallucinating – you can check sources – and then the other team did not. And so that was our single blind kind of test and –

Mr. Allen: You ran it side by side the competition –

Col. Strohmeyer: Side by side. Yeah, exactly.

Mr. Allen: – with generative AI, without generative AI.

Col. Strohmeyer: And the difference was this wasn’t, like, you know, research organizations. This was actual warfighters that were doing this, you know, and seeing what worked, what didn’t work as they went through the process.

So that’s one example of how we’re really partnering with them to get very quick results and to show are we actually testing these things appropriately – are they actually being effective.

Mr. Allen: That’s on the Task Force Lima part, correct? Yeah.

Col. Strohmeyer: Correct. That’s on the Task Force Lima front. We’re looking forward to the – as Alpha 1 scales up this year to start working with them on a lot of those capabilities, bringing those in, because we are partnering with the services and allowing us to be able to test out those capabilities.

Regarding to your question about data mesh and the way that at least on the GIDE side how we are working with warfighters to use those – what we call data mesh services we really view it as two levels.

So the first level what I would call, like, the strategic level one of our missions that we are trying to accomplish there is something we call global integration, the ability for combatant commands and the Joint Staff to have their data in common and to make decisions that are truly globally informed.

If we want to move forces from one location to another they’re not just moving it because somebody requested those forces but because we want to create a global effect by moving those forces. And so bringing all those combatant commands together.

Under that mission, for example, the data mesh services that we are trying to bring to bear allow us to be able to have data in common between the combatant commands so that one combatant command doesn’t have their kind of program of record that they’re working with that has data in a stovepipe and they may have, for example, some logistics data or munitions data that is relative to that force that they have.

Well, in the past that data wasn’t viewable by another combatant command. But now, because we’re trying to truly globally integrate everything we do, a data mesh service in this case that we are working with allows us to have – that piece of data is shared by all the combatant commands, and not just shared via an email or something.

It’s shared live. So they have live access to that data and if one of the combatant commands moves a force it’s represented in that same piece of data for that other combatant command and also available at the Joint Staff. And so that is at that level how we’re using data mesh services to allow us to be able to get better access to data.

The next level for us is what we would say at the operational or tactical level. This is where we really start to overlap with the services where if we’re going to conduct a joint fire mission to be able to close the kill chain the data mesh services there look a little bit different because whereas at the strategic level it’s enterprise level data. It’s usually all available in the cloud.

When it comes to operational and tactical data and decisions that data needs to be highly resilient. It needs to be able to operate in a forward environment and in an environment where it may be contested, where the electromagnetic spectrum may be contested or our access to cloud may be contested.

And so we are using data mesh services for their – like, the joint operating system we just tested it out in GIDE IX to really good effect where we had multiple nodes of data. So rather than a hub and spoke technique of having our data available via a cloud hub and then a spoke at the edge we had a true data mesh deployed where all of the data that was used for those warfighting fires decisions existed on every node and the nodes were intelligently routing the data across this kind of mesh network so that if a piece of that mesh went down it didn’t matter.

The data would resiliently repopulate across the mesh and be able to get that information wherever it needed to go at whatever time, and we tested it to really good effect partnering with the Army at Project Convergence Capstone 4 just about two weeks ago.

Mr. Allen: That’s great. And, you know, you’re talking about the data mesh as it exists at the strategic level, at the tactical level, but really in the operational sort of COCOM-driven world.

But there’s this also aspect of the data mesh and how it exists in the development community, those who are actually creating the capabilities, and to me it seems really important that the data from the operational community be made available to those who are developing the next round of AI models because if you don’t have training data you’re not going to – you know, your machine learning models aren’t going to learn anything without that training data.

So how far along are you in that journey of the actual data that is collected during GIDE actually feeding into the development of the next round of AI models or whatever it may be?

Col. Strohmeyer: And so I think we are on our way. We have a long ways to go. We did successfully just demonstrate in this last GIDE for the first time ever a completely vendor-agnostic data integration layer that allowed us to be able to bring that data down. It didn’t matter what vendor was using it, how they were using it.

That data was created and was extensible across any of the operational systems that we’re using and now it is ready to start piping over into the development pipeline into Alpha 1 and other capabilities so that we can start learning on it at scale as an enterprise.

So that’s one of the things we really want to get after this year is to start making that pipeline permanent, persistent, and real so we can start training those models.

Mr. Allen: That’s great.

And then, Captain Lugo, I wanted to ask you to sort of react to something that Margaret Palmieri, the deputy CDAO at the DOD, said. She said that – earlier this year that CDAO was working to create, quote, “an AI data hub as part of Replicator.”

Can you help us understand, you know, what role your team plays in the Replicator initiative and autonomy efforts, more broadly?

Capt. Lugo: So Alpha 1 has three lines of effort under it. One of them is what we call traditional AI, right? You referenced to that, you know, the typical way that, for example –

Mr. Allen: Supervised machine learning and –

Capt. Lugo: And computer vision and that kind of – that’s line of effort number one. Line of effort number two is generative AI which when we start talking about Lima we’ll – I’ll address that one.

And then line of effort number three is autonomy. The autonomy line of effort is basically AI/ML scaffolding for autonomy. We’re starting to coin now autonomy scaffolding. It’s to support all autonomy projects in the DOD. Replicator happens to be one of them, right? But really because of the need versus maturity – the line of effort number one is the most mature.

Line of effort number two is the one we’re still working with, the generative AI one. The line of effort number three is now prioritizing the sense of need because, really, traditional AI and autonomy have a lot of overlap, probably about – and just making up a number. Don’t quote me. But about 80 percent of the capabilities required for one are in another.

Mr. Allen: Right. Think of a Tesla autonomous car. What makes them that hard? AI is most of the story.

Capt. Lugo: Correct. Right. So they all need data labeling. They all need a model repository and control systems. They all need – both of them need containerization efforts, instrumentation efforts, all of this.

The differences between one and the other is the autonomy line of effort needs modeling and simulation, needs other – there’s other synthetic areas that are required in the – in that line of effort. So that – those capabilities have now been prioritized from our roadmap so that we can support –

Mr. Allen: So you’re developing these modeling and simulation capabilities, or planning to.

Capt. Lugo: I won’t say developing. We are making them available.

Mr. Allen: You’re making available.

Capt. Lugo: Yes. Thank you.

Mr. Allen: This is probably coming from industry, right? Yeah.

Capt. Lugo: (Laughs.) Yes. That is a good way of putting it, right.

So but the other piece – and you alluded to it, so all of this is connected, right, the way we think about these things. The other piece is traditional ways of thinking of autonomy vehicles, again, has been stovepiped, right? You buy a bottle of water but you want to put autonomy on it, right? It’s not how does that bottle of water fit in the autonomy world.

Those are two different ways of thinking about it. So the program offices have been, again, very stovepiped in the way they think of autonomy. As with AI/ML in general we are looking at how can we leverage the similarities of autonomy across weapon systems and more specifically the data.

So if I’m training – if I got data for an aircraft, A, the Navy aircraft will probably benefit from the same data set and vice versa.

Mr. Allen: They might use similar sensors. They might use similar engines. Who knows? Yeah.

Capt. Lugo: A lot more similarity than not, right? And even with terrain vehicles or even surface vehicles can also provide data for other vehicles. So that’s one piece, right? That’s the sensor data and making that data available for all the developers to utilize that.

So you see how the – the theme is converging to the same story here between what we’re talking about. So her point about how we’re supporting Replicator, just to close the loop on that, we are from the perspective of we are enabling autonomy, specifically perception autonomy, and that is – I got to be very specific about that because there’s two pieces of – there’s multiple pieces to autonomy.

Perception autonomy is the closest to taking a sensor and using AI to determine what that sensor saw or interpreted or inferenced, right?

There’s another piece of autonomy which is C2 autonomy, command and control of a particular vehicle. That is still left at the program offices to develop. So that is not what we’re enabling at this time. At this time, what we’re focusing on is on the sensor autonomy piece or the perception of autonomy.

Mr. Allen: But that’s a problem that so many program offices are going to run into. If you can make their life easier –

Capt. Lugo: Absolutely, which is why we’re trying to prompt it, right. A lot of excitement around that technology and, obviously, military personnel were also very excited about it, right?

So as time progressed – and I’m doing a little time line here – from November to about the summer of ’23 – beginning of the summer of ’23 – we started realizing that – we as in DOD started realizing, hey, we need to come up with an understanding of what this technology is in the sense of how can we apply it, should we apply it, and what are the guardrails around it, what are the risks, mitigations, and then also if we’re going to apply it and we decided we are should we do this in what kind of fashion with all the boring terminology. How do we acquire it, what is the acquisition, what’s the sustainment path, all that kind of stuff.

 So all of those are questions that needed to be answered. So a good way of doing that is establishing a task force to address that. Even better was the way Secretary Hicks did it which was, I’m giving you a timeline, 18 months. That’s it. You got to answer all these in 18 months. Otherwise, we’re going to get into rabbit holes and start studying this problem forever.

So in August of ’22 was the actual charter signature of the task force.

Mr. Allen: August of ’22?

Capt. Lugo: No, ’23. I’m sorry.

Mr. Allen: ’23. There we go.

Capt. Lugo: August of ’23. My apologies.

Mr. Allen: Like, did you have an advance notice of ChatGPT, you know? (Laughs.)

Capt. Lugo: No. I did have – I did have two months to set up phase zero of the task force.

Mr. Allen: Ok.

Capt. Lugo: They had a warn order. But the establishment was on August of ’23.

So what have we done, then? We have – again, I like the three line of effort approach, right? So we established who’s who, what are our deliverables, and we also established three lines of effort within the task force. Those are learn, accelerate, guide. Not that GIDE, G-U-I-D-E.

So those three lines of effort are, obviously, ongoing because they’re line of efforts but learn was the one focus in the beginning. The learn line of effort was about collecting use cases from everyone, everything from E-3s submitted use cases to strategic command submitted use cases.

We’ve got now close to 230 use cases – official use cases collected and what we’ve – what we did we categorized those use cases into what capabilities are people really looking for. Yeah.

Mr. Allen: And this is taking the form of, like, a data call to the rest of the Department of Defense asking them, you know, hey, this is generative AI – what could you envision in your mission set using it for? Is that right?

Capt. Lugo: Yeah. Yeah. Correct. We have a portal and people enter those use cases in and then – and that is ongoing. We keep going. And, again, like you said, not just LLMs, which is what people were focusing on in the beginning, which is also why it’s Task Force Lima because when we started we were looking at it mostly from LLM. As we progressed from phase zero to actual charter signature we changed it to generative AI but we didn’t change the name so we kept it as Lima.

However, with that said the generative AI cases are, I would say, that are not LLM, the other modalities, are starting to come in now. There mostly is LLM type – yes.

Mr. Allen: Most of the focus remains on LLMs, yeah.

Capt. Lugo: Correct. Now, we’ve got – so that’s line of everyone so we learned that. But at the same time – and this is what is exciting about this – what we were also learning was what are the actual capabilities out there from industry, and it’s funny because you said do you have any notice about generative AI because I said 2022.

This is the first time that I can confidently say that we are a week to two weeks only behind from what’s actually in the technology space out there and it is because our industry is so forthcoming to us as to, hey, this is what’s coming. This is what we’re doing. What do you guys think, you know.

Not that we’re shaping but they do come in with a way of expressing, hey, is there a use – it’s more of the is there a use case that you could use this in, right, because they know we’re collecting those.

Mr. Allen: So, you know, November 2022 ChatGPT came as a surprise.

Capt. Lugo: Yeah.

Mr. Allen: The first problem that you’ve solved is the DOD is no longer likely to be surprised, right?

Capt. Lugo: Correct.

Mr. Allen: You’ve got the right lines of communication with the right organizations. Yeah.

Capt. Lugo: Absolutely, yes. So we got industry. You know, obviously, academia is also part of this, the FFRDCs. So that’s line of effort number one. Line of effort number two, which is the accelerate line – I’ll tell you in a second here.

Let me just finish with line of effort number three, which is the guideline. That’s the one where we’re going to be writing frameworks, everything from mitigations and risk assessments to, you know, the cybersecurity stuff, the security classification guide, recommendations how to do those.

But we took a line of effort number three item within 30 days of establishment, which was called the interim guidance, to the department. So the interim guidance to the department was like, hey, this is out there – what do we do, right? What are my left and right limits of the tool?

That guidance focused on two main aspects. One of them is all current policies and regulations out there apply.

Mr. Allen: Right.

Capt. Lugo: So just because this is a new tech doesn’t mean that you get to not do what you used to do before, right?

Mr. Allen: And just thinking about what some of those policies might be, right, accessing cloud-based services from the DOD network is not necessarily allowed just because your browser can navigate to that page, right?

Capt. Lugo: Correct. Correct. Exactly. Similar to the internet, right? So it’s all, you know, data that you should not be putting in these models, right? Information –

Mr. Allen: Right. Don’t take your CUI data, copy paste it in the –

Capt. Lugo: Absolutely correct. Yes.

Mr. Allen: – ChatGPT and ask it to do your homework for you.

Capt. Lugo: So that was number one, right. The other piece is a human is always accountable. So don’t point to Google or to open AI and say, well, they wrote it. No, that’s not what happened. You are the author. You are still accountable for any information, any reports, anything that comes out of it and, more importantly, if you’re a commander you’re making decisions based on this. You’re going to be accountable based on that.

So what that did was it opened the aperture to yes, go ahead, experiment and try this. But it still made it query it, be skeptical about it, always have a human in the loop, you know. So that gave us some time to come up with even more focused type guidance, which is that’s what line of effort number three is going to be at the end of the 18 months.

Quickly to line of effort number two, that’s the accelerate line of effort. That one is about building sandboxes, partnering with exercise groups like GIDE and ensuring that we actually now provide more of these capabilities via whatever mechanism whether it’s a license, whether it’s a GOTZ (ph), whether it’s a – you know, a(n) integrated tool into a particular workflow – any of the methods in order to accelerate those use cases that all they need is a little bit more of a push, whether it was they didn’t have the resources together or they didn’t have access to the tools, but all in one of our cages so that we can go ahead and contain those use cases.

So it’s a lot of data experimentation, but in a – with a purpose – experimentation with a purpose, right, which is we are not just doing this as a research project; we’re doing it because we want to accelerate a workflow that is asked by a particular use case.

Finally, what we’ll also do in a combination of line of effort one and two what it does is also – I know that we collected these use cases, which are limited to the imagination and knowledge of the people that submitted the use cases, but there are also some potentials here that may have not been explored, what you all think, and we’ll probably publish that to the force.

So that’s a summary of where we are. I don’t know if you have any particular questions.

Mr. Allen: Well, I’d like to actually turn to you, Colonel Strohmeyer, because you said in the GIDE series you’ve already now got warfighting communities using LLMs in this sort of side-by-side competition.

So you already talked about what the – you know, what that was like. I’m curious, what are the results? You know, how helpful was it to actually use an LLM in this test you ran?

Col. Strohmeyer: Yeah. Yeah. Great question, Greg.

So what we’ve – we found is that for certain use cases that it’s proving to have a lot of value. So it’s not, you know, this panacea but it’s also not this Pandora’s box of evil that we – it’s somewhere in between, right?

And so use cases such as content retrieval when we are – especially when we’re dealing with open source information it’s providing very, very valuable because when it comes to OSINT or open source information there’s just, I mean, terabytes of data that you could be gathering from and it’s too much for a human to make sense of all those things, especially when you’re combining different types of open source information and bringing them together. It just requires a lot of humans to make any sense of it.

But when you take that same information and you give it a very tailored specific prompt about, you know, look for new changes in the road network in this area that may not have been identified previously it can identify that really well and it can also do it in a way that exposes the sources to humans so that they can check and see.

So we’ve seen a lot of very positive responses with that, that it’s that content retrieval but content retrieval in the context of identifying new changes and new patterns. That’s the – it’s one thing to gather content. It’s another thing to then give contextual changes to that content that is really valuable to humans.

So we’re seeing that across many of our workflows that we have inside GIDE, that there’s a lot of promise there. Also, initial content creation. Not the final answer but, like, initial content creation. For those of us that have lived and worked in combatant commands in the past you know that a day-to-day mission that you have – even not just day-to-day but even crisis or conflict is the rapid creation of information for a commander to be able to distill about what’s going on in this area and to be able to give it to them that takes hundreds and hundreds of hours of combatant commands.

And so one of the ways that we’ve seen promise is taking that day-to-day mission that’s required to get that information up there and using things like generative AI to get initial hack at, you know, what’s going on, what’s the information, doing it in a way that we can see what sources were pulled on to be able to generate that information and then have the human kind of curate it and edit it after that. That’s another use case that we’re seeing a lot of promise in.

Mr. Allen: That’s great. And what actually goes into making this available to an operational community?

I think most folks who are watching online have probably played around with ChatGPT. But ChatGPT is, you know, made available in a web cloud interface. That’s not necessarily going to be available to a warfighter if there are degraded communications or, alternatively, it might not just be approved because of cybersecurity guidelines or anything else.

So when you’re using a large language model in the GIDE series what actually goes into making that available to the warfighters? Are you using, you know, a custom system that is being ported onto DOD networks? Are you just allowing them to access ChatGPT? How is this happening?

Col. Strohmeyer: Yeah. No. So we’re being very measured about it. We’re taking our cues from Task Force Lima and the guidance that’s been sent – that interim guidance that X talked about before. But so, one, we are using some things that exist at what we would call impact level four so, essentially, unclassified things.

Mr. Allen: This is the – yeah, the data security – (inaudible) – yeah.

Col. Strohmeyer: Exactly. So we’re making sure that we’re being smart about it because anytime we use one of these things if you use it in certain ways it could be repopulating data back out into kind of the internet.

And so we want to be smart about how we do that so we’re kind of using them in ways, if it’s at the unclassified level, that kind of protects the government’s data and protects our workflows in an appropriate way but still uses all those great things that industry could give us.

But then we’re also using them at the classified way, so up on classified networks that are kind of enclaved off and, importantly, have now access to classified data.

So in a similar way – not the similar tool but in a similar way you would have a ChatGPT that had access to these troves of unclassified data just existing out there commercially now we have in a more protected way access to classified data that can allow the tools to be using those things that are more operationally relevant for our warfighters.

So we’ve been able to give warfighters access to both those and learn as we went.

Capt. Lugo: Yeah. If I could add a little bit.

So the – absolutely. So your point of tools that are persistent, right? Like, for example, some of the common use tools those are not the ones that we’re encouraging right now for utilization, right, for multiple reasons.

Now, with that said, there’s still utility for those, right? There’s a – one of our use cases I keep mentioning a lot is – there’s a sermon writing for chaplains, right. They can go ahead and utilize an open source. No issue.

But the real utility comes into the tools that we can actually isolate into our – so I divide the world into three – the wild, the zoo, and the cages, right? So the wild is what’s outside. The zoo is the building. The DOD is the zoo and then the cages within it, right?

The way that any tool that can actually be containerized and can actually be utilized within either the zoo or a cage those are the ones that we’re contemplating mostly for the use of the warfighters, specifically when you start going to an impact level – higher impact levels, right? There is no way to connect outside.

The only problem with that is that the request is really to get what’s the latest and greatest. The latest and greatest doesn’t perform –

Mr. Allen: Lives in the wild. (Laughs.) Yeah. Yeah.

Capt. Lugo: Lives in the wild. That’s correct. So that’s where we are partnering and putting that as a requirement from our – or more than a requirement, a challenge – a challenge to industry, hey, how can we go ahead, and these tools are in a persistent way in the wild. How can we tame them and put them into the zoo in cages?

So that challenge –

Mr. Allen: Without them getting stale and old quickly, right? Yeah.

Capt. Lugo: Correct. So that’s a challenge that we have put out there and now –

Mr. Allen: Sounds like you’re having conversations with the folks you want to be having conversations with at that point.

Capt. Lugo: Correct. Yeah.

Mr. Allen: Great. So how does – so this capability exists. You know, one, I’m impressed that it’s already working with classified data because we all know the authority to operate process is a formidable opponent in trying to do anything with AI or data in the DOD environment.

But I’m curious. You know, how does this story end in a capability that is sustained and delivered to COCOMs on an ongoing basis? So what has to happen to get from – you’ve run these side-by-side exercises. You have large language model-enabled capabilities that, you know, warfighters love.

Now, how do they get to keep having them after GIDE leaves or when they deploy to, you know, INDOPACOM or wherever?

Col. Strohmeyer: Yes. Great question.

So I’ll just echo what I think the deputy secretary of defense said just a couple weeks ago is that we needed an appropriations bill to be able to do this.

Mr. Allen: That helps.

Col. Strohmeyer: We got that. So that was great.

So now we think in the same way that we had to slow down our efforts because of that we were still – we still delivered them in viable capability, but we did have to slow down considerably. We’re now able to start pushing these things out more to combatant commands in ways that we wanted to.

Mr. Allen: And that’s going to be a CDAO-provisioned capability? Is that the –

Col. Strohmeyer: Much of it will be, but we have a lot of partnerships. So we have partnerships across OSD with other principal staff agencies, with other organizations, with the services. But CDAO does have a unique kind of role and mission in doing that and so we see that as kind of part of that unique role and mission.

Specifically for CJADC2 that is one of our unique roles that we have to be able to give that capability out. So we’re going to be doing that in a way that will look kind of like a program office does, at least in the near term.

But then we also view it this is not – for at least CJADC2 specifically in the AI-enabled capabilities related to that this is not like an end state that we get to where we’re, like, OK, we have achieved CJADC2 – AI-enabled CJADC2.

This is going to be an ongoing iterative effort. It’s going to look like agile software development – that it’s just going to keep developing, keep working side by side with warfighters, but have the programmatic WASTA (ph) behind it that allows us to be able to make it real and make it sustainable at the combatant commands.

But it’s going to look probably different than a normal, you know, hardware program of record will look.

Mr. Allen: That’s great. Anything you would like to add here?

Capt. Lugo: I mean, the challenges are there but also I think there’s a lot of great momentum to solving those. So yes, there are policy challenges. Yes, there are data sharing challenges, especially when you start getting into higher classifications, right?

Mr. Allen: Yeah. But I wanted to ask specifically about, you know, how this becomes a sustained capability. Is that something that should – like, should every program of record be buying a large language model? Should the CDAO be providing that, essentially some other model that I’m not thinking of here?

Capt. Lugo: Right. So that’s – so –

Mr. Allen: This is not yet decided?

Capt. Lugo: – the cheap answer for me to give you is wait until we’re done and then we’ll give you the answer to that, right.

Mr. Allen: Task Force Lima answer forthcoming. OK.

Capt. Lugo: Right. But the – but in actuality it’s somewhere in the middle, right? You don’t want – it’s cost inefficient and it’s unsustainable to say, OK, everybody has their own large language model.

But also the technology is changing so quick that that is not even – that was thought of less – so six months ago. Six months ago everybody was thinking that way. That’s not the approach that we need anymore on this and we’ve all established that foundational models should be built by the companies that build foundational models right now, which is probably a handful, right? I mean, you can count on each hands, right?

Mr. Allen: Yet, CDAO has folks with, you know, relevant skill sets on staff but they’re not going to compete against an open AI or a Google.

Capt. Lugo: There is no point.

Mr. Allen: Yeah. There is no point.

Capt. Lugo: There is no – and not only are they, like, half a billion dollars to build, you got to refresh these things and there’s a sustainability model just with that, right? Now, with that said, there are ways where you can mitigate a lot of the concerns that are out there about building these things in the wild, right, and then maybe build more purpose-built ones.

There’s also methods, you know, whether it’s fine tuning, whether it’s – or other methods to focus – like Nomad was saying, focus models in your particular area in particular. So we’re looking at orchestrations. We’re looking at – so there’s many, many solutions.

I think at the end of the day we’re going to come up with enterprise like solution sets. But I don’t want to even think of excluding any purpose-built models for whatever reason. So it’ll be probably pretty open. I know that was probably the worst answer I gave you right here on that.

Mr. Allen: No. I think it kind of reflects sort of where you are in the process and the fact that, you know, you are – you have your own opinions but you’re also the person who’s in charge of getting the DOD together to have this conversation. So I respect that.

So that does bring us to time and I want to thank you both so much for coming to CSIS and giving us the sort of first look at the Alpha 1 program, the update on Task Force Lima, and the update on the GIDE series.

This has been absolutely terrific. Thank you very much.

Capt. Lugo: Thank you.

Col. Strohmeyer: Thank you, Greg. Really appreciate it.

Mr. Allen: And for those who are watching there’s going to be a brief minute break where we’re going to come back with Jason Brown who’s the lead for applied intuition defense. So if you’ll just wait a few minutes we’ll be back momentarily.

Thanks very much.

 (Break.)

Mr. Allen: Welcome back to CSIS. Once again, I’m Gregory Allen, the director of the Wadhwani Center for AI and Advanced Technologies here at CSIS. And now we’re joined with Jason Brown, who is the general manager for defense at Applied Intuition, or, more accurately, the general manager of Applied Intuition Defense. And previously was a colonel in the United States Air Force, where he oversaw so many different phases of the DOD’s AI transformation.

Jason Brown, thanks so much for being here.

Jason Brown: Yeah, great to be here. Thank you.

Mr. Allen: So I want to start with you and a little bit about your story. You and I had the opportunity to work together at the Joint Artificial Intelligence Center at DOD. But your AI journey at DOD begins even before that. So help us understand where that story begins.

Mr. Brown: Yeah, sure. So I’d spent 26 years as an intelligence officer. So the further you go along in the intelligence career field, it’s less about analysis and it’s more about getting ones and zeros to the analyst, right? So, you know, over the course of time I got really involved in essentially data management, software development, other things that would enable our mission. And then, you know, from there, I did a lot of work with a lot of emerging tech sort of organizations and accelerators. So DIU and AFWERX. And then, of course, Project Maven. So I was the 480th ISAR wing commander. I was actually a customer of Project Maven.

Jack Shanahan, who was – who was leading Project Maven at the time, he and I were – had gone aways back. He was my boss once upon a time when I was a group commander. And essentially, we were trying to find a way to implement AI, computer vision specifically, for the mission to the 480th ISAR wing, which was, you know, focused heavily on imagery analysis, et cetera. And then –

Mr. Allen: Is it accurate – is it accurate to say you were the first customer for Project Maven? Or one of the first?

Mr. Brown: It was definitely one of the first. You know, there were several different customers, different types of imagery that we were focused on. The one that we were heavily focused on for the 480th was something called Gorgon Stare, right? So Gorgon Stare was a pod that went on an MQ-9 that could actually, you know, orbit an area, look at eight kilometers all at once, in high definition. So I had 10 airmen – because that’s – the limitation was essentially, you know, you could have essentially 10 workstations is tied to that imagery.

But if you can look at it, if this table was Mosul, Iraq, for example, and you can only look through 10 soda straws at various different points, you’re missing a whole bunch of intelligence. And so the main effort for Project Maven in support of the 480th was to figure out how Gorgon – how we could apply computer vision toward Gorgon Stare. That actually led us – interestingly, you know, we – it led us to CVPR, the computer vision pattern recognition, conference. Where Jack Shanahan, Drew Cukor, who was, you know, the executive director for Project Maven, and I pitching Google for this particular product. So Google was the performer on this project. I think many of us know, you know, kind of the way that that played out.

I will say that when my airmen were working with Google on this project, they felt like they were part of the project team. And like – or, the product team, right? So they were heavily engaged, you know, the engineers were, you know, at our site at Beale Air Force base. Our airmen would go to Mountain View, and work through it. And so for me that, you know, kind of helped me understand what right looked like. And then as I went to the Pentagon and I worked in the Strategic Studies Group for the – at the time, for the vice chief of staff of the Air Force, I got heavily involved in standing up the Joint AI Center. I led the humanitarian assistance disaster relief initiative. So very focused, again, on computer vision.

So I had essentially, you know, that job, and then started something called the AI Accelerator at MIT. So the vice chief asked me, you know, hey, how do we make AI real? And I said, well, you know, essentially, the Army is talking to Carnegie Mellon. Let’s go talk to MIT. And that – you know, that organization is still going strong. They’ve been able to, you know, put out several different challenges. Some of that tech is now converting and actually transitioning into actual capability. And so, it’s been an exciting journey, right? Just going from the point of, you know, trying to just get software to work to the point that we’re at today, right? And it’s been, you know, really fascinating to see the evolution.

Mr. Allen: That’s amazing. So, you know, you were originally a customer for AI computer vision capabilities, then you became a part of the program that was developing those capabilities, then you became a part of this MIT initiative that was sort of teaching what you had learned to the broader DOD. A really amazing journey.

Mr. Brown: Yeah.

Mr. Allen: And now you’re at Applied Intuition Defense. And tell us a little bit about Applied Intuition.

Mr. Brown: Absolutely. So Applied Intuition is a company that started – so our cofounders are Qasar Younis and Peter Ludwig. They both came from Google. That’s where they met. But more importantly, they came from Detroit. They’re both – they are – you know, their families are both in the auto industry. They essentially, you know, at Google came up with an amazing idea to produce software for vehicles, started Applied Intuition in 2017. And then over the course of time, Qasar, who’s an immigrant, knew that he wanted to give back to the nation. And he knew – both he and Peter – knew that the tech that Applied was developing for, you know, driver assistance in autonomous vehicles would be very applicable to the defense space.

In particular, there was a lot of opportunity to, you know, develop, you know, autonomy and autonomous capabilities for off road. And this created, you know, an opportunity with both DIU, which we’re working on today for the remote combat vehicle, among others. And so we’ve also noted that, you know, kind of working in the defense space, as you can imagine, there are some really, really hard problems. Hard problems around maneuverability and perception that go beyond what you would normally see in a, you know, autonomous driving situation and in the District here, for example.

And so, these hard problems – you know, that our company is about solving these really difficult problems. And by doing that with, you know, our defense customers, we’re able to both take technology that we’ve developed and apply it, but also be able to evolve our technology in a way that both defense benefits and actually it comes back into the commercial sector. So for example, you know, the off-road technology – off-road autonomy technology that we’re developing is finding its way back into construction and mining, just specifically because we got involved with defense.

Mr. Allen: Oh, no kidding? So let’s stick with Detroit for one moment here, because Applied Intuition is getting pretty well known here in Washington, D.C. and throughout the DOD ecosystem. But, as you said, in Detroit it’s, like, a household name. So what exactly does Applied Intuition do for the commercial automotive industry? What does a – what does a GM or a Ford – what do they come to Applied Intuition looking for?

Mr. Brown: Sure. So as you can imagine, everything is driven by software these days, right? So the automotive industry, the OEMs, have recognized that, you know, their ability to provide capability, right, efficiency within their vehicles, you know, safety for the vehicles, mobility for their vehicles, all comes down to a software problem. And so Applied develops that software in a number of different ways. The tools that are needed to essentially develop autonomous software – so everything from mapping. to perception, to data management, to synthetic data, all the way up to actual – the actual autonomous vehicle platform. So and then something in between.

So every OEM is – and Applied works with 18 out of the top 20 car manufacturers in the world. And so every one of them is in a different place with their software journey. Some of them are, you know, much further along. Some of them need, you know, a piece or a part, whether it’s testing or whether it’s actual, you know, components – software components that they want to put on the vehicle. And so Applied is there offering those capabilities. And then, as you could imagine, you know, that is a market that is only growing, right? And so from a commercial sector standpoint.

As we see, as I mentioned, you know, that technology has very direct applicability not just for, you know, ground vehicles. But autonomy all starts generally with perception. So perception drives the rest of that workflow. You know, the capabilities that we develop around perception, you know, are applicable to maritime, they’re applicable to aviation. And wherever you might need an autonomous system, you know, there’s an opportunity to leverage the technology that we’re building in the commercial space.

Mr. Allen: So your company creates the software that allows a car manufacturer or whomever to develop their own autonomy software that actually goes inside the car. So you make the tools that helps the car maker make an autonomous software stack.

Mr. Brown: That is – that is true. And then, you know, but also going all the way through and developing, you know, much more holistic AV stacks, right? So autonomous vehicle stacks.

Mr. Allen: Also to manage that autonomous system and the fleet of autonomous vehicles that are out there in the – in the wider world.

Mr. Brown: That’s right.

Mr. Allen: So that’s amazing. And you said, 18 out of the top 20 car manufacturers in the world are already using Applied Intuition’s software.

Mr. Brown: That’s right, yeah.

Mr. Allen: That’s great. And so now you’ve already talked a little bit about how that comes into the defense ecosystem, but just sort of connecting it to your own experience from your time working in DOD. How would a DOD customer – and you said you’re already a part of the Army RCV Program, among others – you know, what do they get when they work with an Applied Intuition?

Mr. Brown: Yeah. I mean, they get – you know, beyond the amazing tech that we have, you’ve got a company that is hyper focused on solving these very difficult problems. Essentially, you know, the software that goes into some of the things like RCV, the tools that – you know, that is used to develop it, are not far off from the commercial use cases. We see this more broadly with, you know, a lot of AI and autonomy challenges for the DOD.

I will say that there are use cases – I mentioned – I mentioned the off road, and, you know, to some degree, you know, aviation and maritime – that are going to be somewhat unique for DOD. But the foundational expertise, the foundational technical expertise, the foundational software, all of it starts, you know, on the commercial space that you’re able to evolve and modify, and even create new products. But at the end of the day, you know, having an understanding of how you tackle that problem is, you know, what we’re known for, just being able to tackle these very difficult, challenging, autonomous, you know, system problems.

Mr. Allen: That’s amazing. And the Army RCV is, I believe, a robotic combat vehicle, is that right?

Mr. Brown: Correct, yeah.

Mr. Allen: And so that program is still in the development stage, right?

Mr. Brown: It is in the development stage, absolutely.

Mr. Allen: And so your tools are actually helping them develop the sort of initial autonomous system, in this case. But then conceivably you could stay with them even after they’ve completed development, because, as you said, you have sort of tools that are more on the lifecycle fleet management side as well.

Mr. Brown: Correct. And, you know, so much of this is about sustainment, right? And so software is never done. You know, it is – you have to really treat it like a living organism. When it comes to autonomy in particular, you’ve got, you know, cases that you come across – you know, a system will come across – whether it’s, you know, ambiguous, whether it’s a novel, whether it’s rare – and then being able to have that sort of flywheel of development that is taking those, you know, situations, and then updating the software based on what you’re coming across. So it’s never done. It’s not, you know, you put the – you put the software in, and then you, you know, occasionally do a block upgrade. When it comes to especially military equipment, military capabilities, that sustainment has to be constant.

Mr. Allen: Yeah. And you mentioned software is never done, which, of course, was the title of a very famous defense industry board report, coauthored by our mutual friend Josh Marcuse, who’s now a nonresident here at CSIS. But I think software is never done as true of software. But in the case of artificial intelligence and machine learning, it’s really never done. I mean, literally the operational data that you are gathering just in the courses of using your autonomous system, that is the raw material training data for the next update to the overall system. And in a changing world, if you’re not constantly having data that is changing alongside their operational environment, your capability is probably not going to be that useful.

Mr. Brown: Yeah, exactly. And, you know, it’s a constant – you know, the input-output is so dynamic. You’re having software operate in an open system. You need to have that, you know, consistent feedback loop with that new data. I’ll give you an example of how we do that at Applied. I mentioned perception. So perception – understanding perception and performance of perception is so incredibly important.

Mr. Allen: This is, like, sensor data.

Mr. Brown: Sensor data. So whether it be lidar, radar, optical, for example. Knowing how those models – right? So that’s all tied to a model. The model, you know, is recognizing that environment. And so it’s a scenario. So a scenario being, you know, there’s vehicle dynamics, there’s sensor dynamics, there’s the environment, and the actors within it. So the all of that creates data sets that are part of a scenario. And so understanding how perception is performing, how the models are performing in those scenarios, is incredibly important. And being able to monitor that.

So, for example, I’ll use an aerial example. So if you have, you know, a drone that is, you know, not acknowledging, or not recognizing, not detecting certain objects in an environment, and then you can – you can see where that perception – you know, we have tools that can actually see where that performance degrades. And then you can immediately say, OK, well here’s a scenario: The environment, the vehicle dynamic, the sensor dynamic. Where have we seen this scenario before? And then literally going through a library of data and calling up those scenarios, and then creating synthetic, you know, versions of that scenario. And then rapidly retraining the capability to improve your perception very, very quickly.

That is exactly what I mean about, you know, when we – when we talk about how do you sustain software in the field, especially with AI software? You have to be monitoring performance continuously, rapidly retraining, and have all the tools to do the data management, and then the rapid retraining to be able to do that.

Mr. Allen: And that synthetic data, which, you know, that’s a term that a lot of people use in a lot of different ways. But specifically on autonomy, I think what you’re referring to is the ability of, you know, modeling and simulation environments to actually sort of create circumstances that perhaps you – perhaps you only observe them one time in real world conditions, but then you recognize that this is an important scenario that you need to be able to adjust for. So you create 10,000 synthetic versions of that same scenario. And now it’s a part of your training data set. Is that – is that sort of what you had in mind?

Mr. Brown: That’s exactly right, yeah. And, you know, there’s been – there’s been a lot of folks who had been very skeptical about synthetic data. I think, you know, we’ve even heard – you know, senior leaders within the Department of Defense had been skeptical, which is totally understandable. I will say, from my company’s perspective, the automotive industry has to – has to use synthetic data to train, because live testing is super expensive, time consuming, and you just don’t have the iterations of training to be able to do that in a live environment.

So anyone doing anything autonomy with regard to training is using synthetic environment. Not 100 percent, but a large percentage of their training and their modeling is done in that synthetic environment. And so your ability to create those environments, to create those scenarios I described before, is really, really key. And then so you never do 100 percent synthetic. You use synthetic to augment your live training. And then you validate, of course, in a live environment. But it’s critical to be able to do that. because you’re literally iterating, thousands and thousands of times to be able to ensure that that model is leading to a very reliable autonomous system.

Mr. Allen: And I sympathize with Department of Defense officials who might be a little bit skeptical when they hear something about synthetic data, because, frankly, there are companies out there who are making promises they can’t deliver when it comes to synthetic data. But as you’ve articulated it, right, synthetic data is not a substitute for real-world data. It’s not an excuse to not go get real-world data. But it is an opportunity to get vastly more miles per gallon out of your real-world data by extending that dataset, by supplementing it with synthetic.

Because for these incredibly rare scenarios that you’re only going to see a handful of time in real world, well, one data set – you know, one data example is not enough for your AI system, your machine learning system to train on that. It needs 50,000 examples. But if you have the one, you can use synthetic data to create the other 49,000. And thus, this can now be a part of your autonomous system’s capabilities.

Mr. Brown: That’s well said. That’s exactly what we do. And we have to do that for a number of different reasons. But, yeah, that’s exactly what we do. And it’s – so, you know –

Mr. Allen: We actually – I actually recall, you know, a senior executive at a commercial autonomous driving company. They actually said – I think this was in 2017. They said, in terms of delivering performance improvements to the system, they’ve already crossed the threshold where the modeling and simulation-based synthetic data is delivering more performance improvement per month than the real-world data, which they’re spending a lot of money to get that real-world data.

Mr. Brown: Yeah. A lot of money, a lot of time. But at the end of the day, you know, you to your point, you have a lot of rare scenarios, ambiguous scenarios, right? And so you can imagine in the military context with the dynamic, you know, operating environment that warfighters are in, day in and day out, you’ve got to be able to – you know, if you if we’re going to trust these systems, autonomous systems, if we’re going to – you know, if they’re going to be reliable and be used in a way that we can – you know, that can deliver competitive advantage, you need to be able to – you know, they need to be able to constantly evolve and adapt through software sustainment.

Mr. Allen: That’s great. So we’ve been talking about your background. We’ve been talking about Applied Intuition. I want to get a bit of your sense about the DOD’s broader needs when it comes to the types of capabilities that you’re describing. We were just speaking with Captain Lugo and Colonel Strohmeyer of the CDAO. And they talked about modeling and simulation, and capabilities that they want to provide to sort of make life easier for the programs and people across the Department of Defense who are looking to accelerate AI and autonomy. What’s your reaction to sort of that that vision? And how does Applied Intuition fit into the broader picture of, you know, making autonomy development easier in the DOD?

Mr. Brown: So, again, if autonomy is going to be a competitive advantage, the development is the key piece, right? So being able to have a framework. So, scaffolding. We use that term a lot – or, the DOD certainly uses that term a lot. That is that is, I think, a very appropriate term to use. Being able to develop the environment that rapidly, you know, enables exactly the scenarios that I described, sustainment of software in dynamic open environments, that’s exactly what we have to do. And Applied has amazing tools and capabilities along those lines that we, again, developed in the commercial sector and are applying to the defense sector.

There will be some challenges, no doubt, with regard to how you do that. I know there’s a lot of different, you know, either venues or activities within the – within the DOD. The GIDE experiments for example and some others, where they’re trying to really understand, OK, what policies do we change? What capabilities do we develop? What is what is pointing – you know, pointing us in the right way? At the end of the day, I think you have to have a common set of services, and infrastructure, and capability that others can develop on top of.

The challenge, of course, is if you make that overly complex, if you turn it into what I’d like to call a Franken-stack, right? In other words, you have, like, different pieces and parts that you kind of kludge together and you’re constantly replacing those, there’s some disadvantages to that. You’re going to have some inefficiencies. You’re going to have some complexity that’s going to be really difficult to overcome. So the question, you know, that I’m confident, you know, the DOD is constantly asking, is what is the right level of, you know, commonality with regard to capability in sensors and companies? And, you know, and what is the right level of modularity, where we can plug and play and we de-risk certain, you know, scenarios, you know, downstream?

So this is – this is a tough problem. I mean, they’ve got a big job. And, you know, we are definitely there trying to help them through that, especially around – you know, with what CDAO has called the Autonomy Enterprise Platform, which is going to be really fascinating to kind of watch that development of the actual platform that is enabling the rest of the department, bringing autonomous capabilities to bear.

Mr. Allen: Great. And Applied Intuition is exactly the sort of company that DOD leaders have been saying they want to do more work with, right? Folks who have commercial experience in relevant technology domains that are relevant to defense applications. And I want to ask, because you have – through Applied Intuition – this contract that you said DIU was involved with, which is one of those initiatives to try and make it easier for commercially focused businesses to actually start helping the Department of Defense in technology domains. So what has your – what has been Applied Intuition’s experience? What’s been your own professional experience in in these types of initiatives? Are they helping? Is there more that needs to be done in order to bring in more commercial technology?

Mr. Brown: For sure, for sure. I will say, though, that the department has done an amazing job in just – I don’t think there’s any question now that the DOD wants a DIU, with that organization reporting directly to the secretary. That is a strong and bold statement. I don’t think there’s any question, for example, the Air Force wants an AFWERX. They want these technology accelerators because they recognize what – and this is, by the way, you know, not uncommon in the commercial sector, especially large enterprise companies.

What we’ve kind of seen over the course of, certainly, like, the last 20 years, is there tends to be two acquisition pathways for companies that are – that are, you know, really understanding that software is going to be critical to their competitive advantage, to their survivability, et cetera. Software and other types of emerging technology that, you know, tend to follow a different acquisition path than maybe, you know, the hardware and the other components that actually make up an enterprise, you know, capability.

So traditional acquisition pathways are still needed. They’re especially needed to scale capability and integrate a lot of other aspects – doctrine, personnel, training, et cetera. But you’ve got to move fast on the emerging tech side. You’ve got to move fast on the software, you know, development side. That is where this kind of alternate pathway and these organizations, like DIU and AFWERX, for example, have proven so critical. That, you know, for my company in particular, has been absolutely, you know, critical for us to actually – just be able to get connected with the warfighter, be able to provide this amazing technology that we’ve developed for the commercial space, and bring it very quickly.

I can’t imagine that following the same pathway through a traditional – you know, I think we’d get there eventually with, you know, a program of record and some other things. But this kind of first entry with DIU, through other transactional authorities and other contracting methods that are that are nontraditional, is critical.

Mr. Allen: And so it sounds like that’s your goal, right? DIU is the front door, almost, for the Department of Defense. But for this to be a story that ends in real sustainment operations at scale, it has to be the front door to programs of record, right? Which really have the budget, have the sustainment, and have the man, train, and equip, you know, mandate.

Mr. Brown: Yeah, that’s right. I mean, you know, that is fundamentally how the department is designed. That corporate process is critical to be able to scale those capabilities and ensure those capabilities are enduring. But they don’t lend themselves initially to really bring in that emerging tech. So once the emerging tech gets into that space through this alternate pathway I described and that the department has developed, that’s exactly right. It can scale and benefit the warfighter, you know, in a much bigger sense.

Mr. Allen: And then that – we were just talking about organizations like DIU and programs of record. The sort of other side of the equation is what Colonel Strohmeyer was just talking about previously, which is GIDE, which does not refer to itself as “exercises.” They’re an experiment series. But there’s things like GIDE which are bringing in the combatant commands actually giving warfighters an opportunity to try out these new types of technologies. Other examples would be Task Force 59, which has been part of CENTCOM and the Navy’s work in this similar area. So I’m curious, you know, how does the combatant command outreach initiatives and the commercial technology adoption efforts at combatant commands – how does that affect somebody like an Applied Intuition? Is that sort of thing useful to you? Is it attractive to you, as a commercial company?

Mr. Brown: Yeah, absolutely. I mean, these are lighthouse, you know, sort of initiatives that enable the department to kind of ask smarter questions, be able to really start to explore where the – where the technical and policy debt exists that prevent them from actually developing and moving out on capability. But, you know, the challenge, of course, is you have to be very careful with any – you know, and I haven’t participated yet in GIDE. I’ve been watching it from the sidelines. I think it’s fantastic that the department is moving out on these experiments.

You always have to be careful, though, that you aren’t biased toward one capability or one technology or, you know, another. That you’re actually, you know, focused on exposing the problems and exposing the issues that you’re trying to tackle. And, you know, at the end of the day, you’re making policy and investment decisions. And then you also have the big and key questions like, you know, does it work? Does it work better than what we had before? Are the users, are the warfighters, benefiting from it? And, for all of the above, how do we know, right? And so I think these sorts of efforts and exposure to COCOMs and others who have a very, you know, on the ground, you know, warfighter sense, is really key for a company like ours to be able to, you know, connect and understand what problems we should be solving.

Mr. Allen: That’s terrific. And you mentioned the tech and policy debt. And technical debt is a term of art that everybody in Silicon Valley knows about. Policy debt is not something I hear too often in Washington, but I already love this term. So can you talk a little bit about, you know, what are the technical blockers, what are the policy blockers that make it difficult for the DOD to sort of accelerate its adoption of AI and autonomy the way it wants to?

Mr. Brown: Yeah. On the technical debt side, I mean, it’s well known, I think, at this point. It wasn’t necessarily well known, you know, maybe a few years ago. It comes down to, you know, the first issue is data management. How do you get the data that you need? And how do you – and so that also points to an infrastructure challenge which, again, technical debt. You know, when the – when the pandemic was kicking off, the vice chief asked me – he said, how is this going to affect our readiness? And I looked under – you know, under the hood, if you will, of various different – so I looked at readiness for the United States Air Force, kind of it was – OK, it was a – for a pandemic – was a personnel problem, was a medical problem, was an operational problem, was a logistics problem.

And as I looked under each one of those – you know, or opened each one of those doors, if you will, around the whole data management piece, it was not pretty, right? It was loaded with technical debt, at the time. And they have made a lot of progress since then. And this is just 2020. At the time, it was literally 359 separate systems and databases in Air Force logistics and maintenance, half of which were written in COBOL. I mean, you cannot get to AI if you have this type of situation. So there’s a lot of that – a lot of that that has been, since then, recognized and overcome, and a lot of positive momentum on there.

On the policy debt, boy, there’s all kinds of, you know, challenges. I think one of the –probably one of the biggest is authority to operate and the ability to actually get accredited –software accredited and technology accredited to use.

Mr. Allen: Yeah. And the ATO process – this is the mechanism by which software gets permission to touch DOD networks. It’s principally a cybersecurity-driven process, and a review of that. But as an empirical outcome, you know, we’ve heard folks, including folks from the CDAO, to say that the ATO process is among the biggest blockers in accelerating DOD AI adoption. Just talk a little bit about your experience with this.

Mr. Brown: Yeah. I mean, well, that experience goes all the way back, you know, when I was actual, you know, in uniform, you know, in my wing. Wanted to get capability to my airmen. Again, back to the – getting ones and zeros to the analysts, right? I knew that there was software capability that I wanted them to have. I knew that there was development environments that I wanted them to have. It was remarkable for me to watch just so many airmen actually developing code, right, who are not, you know, by any stretch of the means, trained software developers. But they figured it out. They were trying to overcome a pain point that was, you know, critical to them providing analysis. But the ATO, the – you know, the – essentially, the accreditation process to get them those tools and capabilities was just so long and cumbersome.

I will tell you this, if we want to use LLMs in a really effective way, like, use them to actually write a system security package for an ATO –

Mr. Allen: (Laughs.) And this is like a phonebook-thick, you know, document that you have to fill out basically in paper saying: Please let my system touch the –

Mr. Brown: It’s a paper – it’s a – exactly. So but imagine that just for a second, like, that, you know, you have – you have all of the guiding, you know, regulations, right? You know, think it’s NIST 800-53, and I might have messed that up, but a bunch of these things. You have a bunch of system security packages that have been approved and accredited that can be loaded as your – you know, as you’re fine tuning an LLM. And imagine, you know, you engage that LLM with this new situation. And then you can actually get, you know, an LLM to build that SSP out for you. Walk it through. And you reduce the pain points on the authorizing official, the accreditation authorities. On the person who has to actually develop thing. You also, as kind of a byproduct, actually, like, standardized these things.

But anyway, but the policy – the policy, debt around that particular issue, I think, is one of the biggest blockers. And work – the department has to figure out how to adjust that policy to, obviously, you have to have secure systems. And, you know, interesting, we’re not talking enough about AI security. It’s something we talk a lot about in my company, and in my last company, you know, for that matter. And so –

Mr. Allen: Yeah, because there’s Chinese cybersecurity firms that are bragging about their ability to hack autonomous cars. Presumably, they’re thinking the same thing about military autonomous systems.

Mr. Brown: Yeah. Now, we, obviously – you know, that – security is life or death for a company like ours, and every other software company out there. You know, when I was at Google, we were – we were talking heavily about, you know, AI security and figuring out what is different from AI security compared to normal software security. So that is all needed to be accounted for. However, we’ve got to figure out a way to streamline this process. You know, otherwise it continues to be a barrier not just for companies who are trying to do business and get their capability to the DOD, but, you know, for the – for the warfighter themselves.

Mr. Allen: Mmm hmm. So one of the more exciting initiatives that’s come out of DOD in the past year or so is what Deputy Secretary of Defense Kathleen Hicks announced, which is the Replicator Initiative. Which, you know, in her words, aims to field thousands of low-cost, attritable autonomous systems by August 2025. So in the not-too-distant future. And I’m interested here, because this is one of the most high-level DOD officials to very publicly endorse autonomy and autonomous capabilities as foundational to the future of military power and warfighting. And when you see that initiative, you know, what’s your view of the Replicator Initiative, and its implementation so far?

Mr. Brown: That’s a great question. First of all, I’m glad it’s a thing, right? I’m glad. I’m glad there’s, you know, a very clear impetus on developing autonomous capabilities and a recognition, especially, you know, as we as we look at Ukraine and we’re starting to infer some lessons from that conflict. You know, Replicator, from what I know, to your point, it is about attritable autonomous systems. It’s also about an acquisition accelerator, from what I’ve heard the deputy secretary say. And I think it’s also very much evolving. So I think the – you know, it’s really important, I think, for the department to continue to, you know, refine and communicate, you know, its perspective on Replicator and what does that mean.

Is that a pathway? Is there something we should be doing in industry that is very focused around solving a very specific problem? Because we’re ready to do that. We just want to know, you know, with as much clarity as the department can provide what those problems might be. To be fair, that’s going to evolve. The department’s going to have new considerations. There’s going to be some things that come out. And that’s totally fine. As long as that communication keeps going.

The other thing is funding. You know, originally Replicator was not funded – or, not asked for funding for FY ’24. The latest appropriation, I think it’s 200 million (dollars) is going to the Replicator program. I imagine there’s going to be a lot more coming in FY ’25. It’s great. But as we could imagine, you know, with an appropriation for Replicator, what is – what problems is that, you know, appropriation intended to solve? And then that will help us, you know, in industry really focused our development.

Mr. Allen: I think the other encouraging thing that Deputy Secretary Hicks said recently – and she’s actually echoing words that I said at the last Nexus Summit – which is, you know, if we want more autonomous systems we can think about increasing the water pressure – which is, you know, adding more funding – or we can think about unkinking the hose. And, you know, the kinks in the hose – so there’s technical debt, there’s policy debt – that makes it so difficult for the folks in DOD who want to buy, or build, or whatever AI to actually do so.

And I think the fact that she’s talking about these kinks in the hose, she’s talking about working through this policy debt, I think, hopefully shows that not only is there this $200 million for Replicator coming, or these other programs of record that have announced goals of implementing autonomy, but there’s also an effort to get after ATOs and these other policy and process blockers that make it so difficult to use the money that you do have, when you’re fortunate enough to have it in the DOD.

Mr. Brown: Yeah. You know, the kinks in the hose, I think, is a fantastic analogy. You know, and I think, you know, simultaneously you want capability, so – I believe it – I believe you also said, you know, your options are, you know, undo the kink in the hose or just really turn on the water pressure and just try to – just try to crank through, you know, the kink itself. You have to do both, right? You’ve got to figure out a way to –

Mr. Allen: Especially on the timelines we’re trying to operate on.

Mr. Brown: A hundred percent, yeah. A hundred percent. So, you’ve got to be able to move fast, start acquiring. You know, there’s something to be said about – you know, going back to Replicator – let’s do this thing. Let’s move fast, and then expose the kinks, shine a light on the challenges. And then that gets us very focused on where do we – where do we address it? Going back to my readiness example, right? I want to – I want to be able to know, from – you know, going back in time, this problem that we were trying to solve with the pandemic. You know, how does it affect our readiness?

The question is, what – you know, ultimately, we should know what activities we want to start, what we want to stop, with the aircraft availability will be, et cetera, et cetera. These are the real questions that we’re trying to get after to go after readiness. You know, and then, fundamentally, you know, you start down this path, you expose new problems, like 359 separate systems and databases. But when you move out, you know, which Replicator is an example of, right, you’re going to have that opportunity. Learn new things about where those kinks are, and then how do you fundamentally address them.

Mr. Allen: And for somebody like Applied Intuition, whose, you know, bread and butter is the automotive industry, when she talks about low cost, attritable systems, is the tooling and the development, you know, tools that somebody like an Applied Intuition would provide, or the fleet management, lifecycle management tools that somebody like an Applied Intuition would provide – how are they different from something that’s like the Robotic Combat Vehicle, which is meant to be an enduring, pretty expensive platform, versus something that’s, like, low cost and attritable, like the quadcopters we see in Ukraine, which, I think, has been a part of what Deputy Secretary Hicks is talking about for Replicator. What’s the difference?

Mr. Brown: Yeah. Well, let me talk about what’s the same first. You know, this belief, I think, that, OK, I’ve got a low-cost attritable system, and I can it, you know, autonomous, and you know, it’s going to go out there, and it’s going to find a target, and it’s, you know, going to accomplish its mission. It still has, you still have reliability, you know, challenge, right? How do you know, this is going to act in the way that you want it to, if it’s going to be reliable?

Mr. Allen: Because, you know, a $1,000 quadcopter crashing might be cheap. But a $1,000 quadcopter crashing into your $30 million helicopter is not cheap, right? You know, these things need to be reliable.

Mr. Brown: A hundred percent. And, you know, it needs to be able to do its mission. It needs to be, you know, interoperable. You know, it needs to be, you know, something that you are constantly testing and validating. It’s not like you show up at a battlefield with a bunch of things that you just developed and then, you know, employ them. There’s still very much a test and validation process that has to go into that. And it also needs to be continuous.

There’s going to be – you know, we’ve seen, for example, the Russians, you know, doing various different things to try to, you know, address, you know, various different algorithms, computer vision challenges. I mean, you know, put tires – I think they were putting tires on their bombers, or something – something akin to that, right? They were – they were – you know, they were finding a way to build a better mouse, you know, to defeat the mousetrap.

And if you’re not, you know, constantly updating and sustaining, you know, per our earlier discussion, even if it’s low cost and attritable, they still need to be reliable. That’s still very much, you know, a test and evaluation. That’s still building tools and a workflow that is constantly updating that software that is embedded in those systems. For sure, though, you know, there’s a lot of things like, you know, a Remote Combat Vehicle. One of the things, you know, of course, you still have a human who is – who is connected to that Remote Combat Vehicle.

And then so there is going to be some level of human interaction and human in the loop, and/or how users are actually engaging with this system. And understanding that dynamic as well, right? What is – what is the data that we can, you know, pull from that interaction between the human and the autonomous system that we can understand how humans are interacting with the system? Where does this system provide value? Which scenarios is it really useful for? What scenarios does it need more updating and more development?

Mr. Allen: And you mentioned test and evaluation, which is one of the most important, you know, parts of the autonomy lifecycle, and also one of the ones that Applied Intuition and really specializes in. So can you talk about a little bit – and you’ve already touched on this – but just a bit more on sort of what are the unique features of the test and evaluation challenge in the autonomy context? And then also, like, what Applied Intuition specifically does to address those types of challenges?

Mr. Brown: Sure, absolutely. So I go back to that example, with being able to rapidly collect the data from the scenarios that you’re – that you’re, you know, engaging in and that you’re coming across with regard to – in the case of automotive, it’s, you know, the scenario of, you know, driving down the street. You know, you’ve got potentially a bunch of folks crossing the street on their bikes. And, you know, that scenario needs to be replicated. And then, you know, it needs to be very rapidly brought up. And that whole performance of that entire stack needs – in those scenarios – needs to be evaluated and adjusted based on how it’s currently performing.

You could imagine, though, the cases of mobility for – whether it be aerial, maritime, ground based autonomous systems for the DOD. There’s a lot more engagement with a lot of different actors and environments and scenarios that are very complex. So the toolset that we’ve developed originally for those automotive scenarios really comes down to understanding that performance of the entire stack, understanding how we recreate those scenarios or pull those scenarios from data that we’ve collected. And then, again, really training against them to be able to ensure that there’s this constant sustainment and feedback loop going into the software.

Mr. Allen: That’s great. So we’re coming up on time here. And I want to get your sense of what the future holds for Applied Intuition and for, you know, the DOD AI and autonomy transformation. So you’re a relatively recent addition. This is your first year running the Applied Intuition Defense team. And what are you excited about over the next 12 months for Applied Intuition?

Mr. Brown: Oh, I mean, that’s – where do I begin? So the amazing – so, we’re kind of in this – what I compare airpower was in the 1930s, right? So there was some great demonstration of capability. And the operational concepts of airpower, you know, were being developed in that time. And then it was put to – you know, it was scaled in the 1940s. And then, you know, I think that’s where we’re at with autonomy today. Both the technology is really maturing incredibly quickly, how we actually, you know, interact with it, how we develop it, the operational concepts that we’re, you know, beginning to develop to be able to scale that capability.

Applied Intuition is right in the middle of all of that, right? To be able to, you know, both take the technology and the operating concepts to figure out how we do this at scale. One of the other things is, how do we do this not just at scale for a very particular capability, but how do we do this at scale for all autonomous capabilities? And so I mentioned Autonomy and Enterprise Platform before, that the CDAO is developing. I think that is a very exciting opportunity for – you know, for the department to actually, like, develop a foundation that other autonomous capabilities can plug into. So that, I suspect, is going to keep us very busy over the next year.

You know, we are – we have every year, as you well know, you mentioned Nexus before. On June 13th we’re having our Nexus event, which is going to be at the Kennedy Center. Where we’re going to be talking about all of these things. We’re going to actually showcase some of the technology, the technology that is going into these, you know, autonomous systems, and then discussions around the implications of all of it, you know, from a national security perspective, you know, from a warfighter perspective. It’s going to be really exciting.

Mr. Allen: I had heard Nexus was coming back, and I was really excited for that. But I hadn’t heard it was at the Kennedy Center, which is going to be incredible. And so I can’t wait to attend to myself.

Well, Jason Brown, who it’s very difficult for me to not call you colonel – (laughter) – but, Jason Brown, so glad to have you at CSIS. Thanks very much for taking the time.

Mr. Brown: Thanks so much, Greg.

Mr. Allen: Well, this concludes our event. And thank you all for watching so much. Later this week, we have another event on DOD’s AI adoption. We’re talking with Matt Turek of DARPA about DARPA’s vision for artificial intelligence and autonomy. Thank you all for watching.

 (END.)