The Collection Edge: Harnessing Emerging Technologies for Intelligence Collection
July 13, 2020
- U.S. intelligence collection organizations can leverage emerging technologies such as artificial intelligence (AI), advanced sensors, cloud computing, and advanced analytics, to improve how intelligence is gathered, processed, and exploited for operations.
- These technologies could enhance and automate a variety of core collection and processing tasks across intelligence domains and enable collection to better adapt to changing adversary behavior and operating environments.
- The fielding of “edge” sensors and processing devices with AI and cloud computing could transform where, how, and how fast intelligence is collected to drive operations and decisions.
- U.S. adversaries will also be innovating rapidly to develop, field, and exploit tech-enabled intelligence tools to degrade and disrupt U.S. and allied collection.
- In the competition for the technological-intelligence advantage, the United States will need to accelerate the speed and scale at which technology is acquired and integrated and the skillsets and missions of intelligence professionals are adapted in order to stay ahead.
Generating timely, accurate, and insightful intelligence for U.S. national security leaders often begins with intelligence collection. In a world of proliferating sensors and big data, the U.S. intelligence community (IC) holds unprecedented capability to collect information of national interest—in more places, through more means, and at greater speed and scale. However, while the tools and resources available to U.S. intelligence collection continue to grow, so too do the challenges.
First, data is not intelligence, and the sheer volumes of it collected—classified and open-source—far exceeds analysts’ ability to process and turn it into meaningful intelligence and insights. Second, collection assets—human and machine—need requirements and priorities to steer collection. But the increasing number, diversity, and rapid shifts in intelligence targets and security threats make prioritization difficult and can quickly render well-laid planning obsolete. Indeed, an IC entering 2020 already attempting to balance collection on great power rivals, regional adversaries, terrorism, cyber, and disinformation, has had to swiftly shift focus to a new target, pandemics and global health, as Covid-19 reorders intelligence priorities.
Amidst evolving threat environments and changing national security priorities, U.S. and allied intelligence collection organizations can leverage emerging technologies to improve how intelligence is gathered, processed, and exploited for operations. But how? In part one of its three-phase study, the CSIS Technology and Intelligence Task Force explored how advanced technologies, such as artificial intelligence (AI)i and its subset, machine learning (ML),ii advanced sensors, cloud computing, and data analytics, can empower U.S. intelligence collection. The Task Force’s core research question was what are the opportunities and challenges posed by emerging technologies to intelligence collection, including geospatial (GEOINT), signals (SIGINT), and human (HUMINT) intelligence? To answer it, the Task Force convened experts and stakeholders from across the IC, technology, policy, and research communities for a series of discussions.
This CSIS research brief summarizes the main findings from the first phase of the Task Force. The brief begins by studying the near-term ways technologies can be integrated into collection, processing, and other intelligence operations and how well the IC is utilizing these capabilities. It then assesses the key threats and challenges the IC faces in adapting new technologies into the collection mission. The brief concludes by exploring the implications of technological transformation for the IC and the over-the-horizon technologies likely to impact future missions.
The Task Force identified opportunities for leveraging advanced technologies to improve, accelerate, and augment how intelligence is collected. Some applications are already being integrated into IC collection and processing tasks and can be further enhanced, embedded, and dispersed across IC organizations. Other opportunities are emerging technologies identified as having intelligence mission value in the coming years.
Emerging technologies, particularly AI, advanced sensors, and big data analytics, could enhance and automate a variety of core collection tasks across the intelligence collection disciplines (INTs), including collection management, signals detection, and target identification. Automation and ML can help humans establish baselines of normal behavior and activity, alert humans when those patterns deviate, and adapt collection to changing adversary behavior and operating environments.
Collection Management: For the technical INTs, AI tools can assist in automating the planning, scheduling, and tasking of collection platforms and optimizing asset selection based on collector’s requirements and type of target. AI can assist to forecast collection tasks, select the best asset for the objective, and schedule missions based on task frequency, range, and scope.1 Advances in deep learning could enable more automated decisionmaking and adaptive tasking of collection assets. As AI models learn how analysts tasked and platforms searched and identified targets in differing operating environments, the day-to-day management and tasking of technical collection could require less human involvement through such reinforcement learning.2 The end result for the IC could be collection management systems centralized at the requirements level based on national priorities but decentralized at the target and asset selection level. This could enable collection posture to be more adaptive to operational environments and smarter in allocating high-end platforms.3
Automation and machine learning (ML) can help humans establish baselines of normal behavior and activity, alert humans when those patterns deviate, and adapt collection to changing adversary behavior and operating environments.
Signal Detection and Early Warning: Technical collection can harness advances in advanced sensors to detect more types of enemy signals, identify imperceptible changes in target environments, and sense anomalous or high-risk behavior, providing enhanced “early warning” for analysts. Forward deployment of multi-mission smart sensors could enable greater penetration of denied areas and collection on hard targets. Multimodal fusion from individual sensors and networked systems could enhance data capture and sensemaking of signals from multiple INTs, using signatures as varied as thermal imaging and vocal inflection to detect targets.4 As technical collection capabilities mature, data scientists and analysts can work to build, test, and hone AI models of likely enemy signatures, patterns, and activities and smartly search broad areas for priority targets. Such “smart search” capacity will be critical as 5G and the internet of things (IoT) devices generate exponential growth in signals and data.5 Such AI-enabled signal detection and search models could be integrated into indicators and warning systems automated to “tip and cue” collection.6
Target validation: AI and advanced analytics could be used not only to identify collection targets but also to validate the veracity of collected data and collection sources, technical or human. AI is already applied in automating aerial reconnaissance, surveillance, and target monitoring, and ML algorithms are used to verify and validate targets collected from imagery and full motion video. AI tools and analytics can also be exploited to enable HUMINT specialists in their core mission: identifying, recruiting, and securing intelligence from foreign agents.7 ML algorithms could be trained to “spot and assess” potential agents by combing open-source data. Advanced analytics and facial recognition can then help construct “digital patterns-of-life” of recruitment targets, assisting in predicting their activities and verifying their access to desired information.8
AI and advanced analytics already assist the IC in processing exploding data streams into digestible and usable information and could improve how data is streamlined, curated, and prioritized for analysts. Building off foundational achievements, such as the application of computer visioniii to imagery and video processing and natural language processingiv for text extraction, these tools could enable greater automation of more complex processing and frontline analytic tasks. Advances in machine sensemaking of data, including knowledge extraction and visualization, hold the greatest potential for automating and offloading vital but time-intensive human tasks to machines, creating more bandwidth of collection agency analysts.
Triage and Notification: The most immediate application of AI for processing is helping to “triage” and sort the IC’s massive data and information flows, automating tedious and time-intensive tasks still often done manually.9 ML algorithms could be honed to scan and comb large datasets, such as from imagery and SIGINT collection, for information prioritized for specific analysts.10 AI could augment existing techniques such as batch processing to deliver bulk search results to analysts in digestible and usable forms.11 AI could also assist in integrating processing into indicators and warning frameworks; as data is processed, AI tools could be trained to spot and flag information designated as critical and send automated alerts to the analysts and decisionmakers.12
Task Automation: AI is currently used in automating processing and front-end analysis of sensor-derived technical data but has growing potential to take on more cognitive tasks for analysts. For GEOINT, AI assists in processing and labeling imagery and full motion video collected from satellites and aerial ISR platforms. Advances in deep learning neural networks could help automate more complex tasks in image recognition and categorization and at greater levels of precision and specificity for users (e.g., not only recognizing a tank but identifying its make, model, origin, and, combined with other signatures, recent activity or deployments).13 For all technical INTs, but particularly SIGINT, NLP was identified as a vital and value-added AI capability.14 NLP applications for the IC are evolving from speech-to-text transcription, voice identification, and foreign language translation of intercepted communications to more sophisticated, human-like capabilities such as narrative text summarization and emotion detection.15
Sensemaking and Visualization: In the coming years, analysts could employ deep learning tools to identify patterns and trends in data streams, make inferences on relationships between targets, and visualize networks for enhanced clarity and deeper meaning. Experts identified graph analytics and the applying of neural networks and other ML techniques to graph data—data that can be connected, like people, organizations, and locations, as nodes in a network—as a key area of development. Algorithms could be able to classify, cluster, and delineate data into nodes; make inferential judgments on the nature of the connections between nodes; and generate visualizations of key patterns and influence networks.16 As these AI tools progress, the back-end result of better data processing could be better and automated data sensemaking delivered in digestible and actionable forms for both collectors, analysts, and decisionmakers.17
Advances in machine sensemaking of data, including knowledge extraction and visualization, hold the greatest potential for automating and offloading vital but time-intensive human tasks to machines, creating more bandwidth of collection agency analysts.
While the Task Force’s primary focus is strategic intelligence, this research phase also revealed opportunities for applying emerging technologies to key operational dimensions. Tech advances could reshape how HUMINT operators recruit and handle agents. The fielding of “edge” sensors and processing devices with AI and cloud computing could transform the speed and scale of tactical intelligence collection to drive rapid operations. In short, technology could push strategic and tactical collection forward, enabling collection of hard and time-sensitive targets and reducing user demand—whether it’s from analysts or operators—for national or more exquisite systems.
HUMINT Operations: Core HUMINT missions—recruiting spies, collecting intelligence, and conducting covert action—will endure, but emerging technologies could transform the tradecraft and tools to execute them. While personal interaction will often be crucial in a source operation, agents today can be spotted, assessed, developed, recruited, and handled virtually. Advances in encryption and device computing power could make source-handler communication more secure and robust in compressed time windows, while AI-enabled surveillance and analytics can improve monitoring for security and counterintelligence risks.18 The digitization of secrets sought by the IC is blurring the traditional boundaries that separate the HUMINT, SIGINT, and cyber disciplines—and the agencies organized around them. The blending of technical tools with collection missions, such as HUMINT officers using SIGINT tools, could enable more penetrating foreign intelligence collection and more creative covert action, political warfare, offensive counterintelligence, and other sensitive intelligence operations.
Edge Collection: Advances in purpose-built edge devices and cloud computing could help transform where and how fast data is collected, analyzed, and acted upon in forward operating environments. When paired with advanced sensors and communications platforms, human collectors equipped with edge devices could push into harsh, high-risk, or denied areas for sensitive collection missions and transmit time-sensitive data in close to real time.19 As edge capabilities mature, AI-enabled processing tools could be applied in forward areas or even on-site, rapidly generating actionable intelligence for operators and decisionmakers.20 While expanding where and accelerating how fast intelligence is collected and actioned, edge devices could help alleviate operational demands on national or theater-level collection platforms.
Exploitation and Actioning: AI applications can accelerate the speed and precision of sensitive site exploitation (SSE) of captured enemy materials and shorten the processing, exploitation, and dissemination (PED) cycle for intel-driven tactical operations. AI-enabled tactical forensics tools can assist in processing massive amounts of digital materials from captured devices, filtering and extracting prioritized data, such as names, phone numbers, or images of specific people. Automation tools and integrated APIs can then dispatch specific data to specific receivers in the intel-ops cycle, enabling deeper analysis and/or more immediate actioning by operators.21
Advances in purpose-built edge devices and cloud computing could help transform where and how fast data is collected, analyzed, and acted upon in forward operating environments.
As U.S. intelligence moves to integrate emerging technologies into collection operations, its global competitors and foreign adversaries will also be innovating rapidly to develop, field, and exploit tech-enabled intelligence tools to degrade and disrupt U.S. and allied collection. In the iterative competition for the technological-intelligence advantage, the U.S. IC will need to accelerate the speed and scale at which technology is acquired, data solutions and architecture are adopted, and skillsets and missions of intelligence collection professionals are adapted in order to stay ahead.
Adversaries and Threats
The proliferating number and types of national security threats facing the United States and its allies have already strained the IC’s capacity to collect intelligence across the threat spectrum; adversaries’ use of emerging technologies to directly target and degrade U.S. collection could strain and potentially weaken U.S. capabilities even further. The same technological tools augmenting U.S. intelligence will empower and embolden foreign intelligence rivals—namely China and Russia—in detecting, denying, disrupting, and deceiving U.S. intelligence collection efforts. As one Task Force participant noted, the IC will likely face a “vanishingly short shelf-life of secrets,” as its ability to keep officers, operations, and information secret—as well as its monopoly on being able to collect that information—will likely diminish.
Detection: Accelerating use of smart sensors, surveillance, and biometric tools will transform operating environments into ones of “ubiquitous surveillance”—not just in authoritarian states like China and Russia but even in neutral or allied countries across the globe. As high-counterintelligence threat areas expand, case officers will struggle to maintain cover and operate clandestinely, facing a persistent risk of exposure—of themselves, their agents, and their operations. As technologies weaken defensive counterintelligence capabilities against foreign detection, they will also strengthen adversaries’ offensivecounterintelligence capabilities to penetrate U.S. intelligence. An ever-growing “digital dust” of publicly available information combined with big data analytics and targeted hacks could enable foreign intelligence to identify and expose U.S. intelligence officers, including at home.22 Foreign case officers will exploit the same tools to speed up the agent acquisition cycle, including of U.S. spies, while foreign AI-enabled technical collection targets U.S. operations and activities.
Denial: The proliferation and integration of emerging technologies into adversaries’ collection platforms and tradecraft will make already “hard target” actors such as China and Russia more impenetrable and could create more “denied areas” for operations. AI-enabled advances in cybersecurity and cryptography will help adversaries to harden and encrypt their systems and complicate U.S. efforts to penetrate and collect on their networks.23 For HUMINT collectors, intensifying and sophisticated hostile surveillance from intelligence services could provide the host-nation persistent, blanketed coverage of U.S. officers, complicating traditional agent communication techniques.
The same technological tools augmenting U.S. intelligence will empower and embolden foreign intelligence rivals—namely China and Russia—in detecting, denying, disrupting, and deceiving U.S. intelligence collection efforts.
Disruption: In addition to targeting human intelligence officers, hostile foreign intelligence services will likely also exploit AI-augmented tools to put technical platforms under persistent threat by penetrating, manipulating, and degrading collection. AI-accelerated cyberattacks could target collection and communication platforms and employ intelligent malware to access, exploit, or destroy critical data and intelligence.24 Once inside, foreign intelligence could exploit “counter-AI” techniques to insert “poisoned” or false data into training sets to undermine ML-based models, fool U.S. IC algorithms, and cause AI systems to misperform—such as a deep neural network image classifier falsely recognizing friend as foe.25
Deception: U.S. adversaries will attempt to use AI tools not only to deny and disrupt U.S. intelligence but also to deceive it and flood the U.S. collection enterprise with disinformation. Deception techniques to fool algorithms into misclassifying data and the use of generative adversarial networks to create “deepfakes” of imagery, communications, and intelligence reports could be honed and scaled up over time to overwhelm and confound collectors and analysts.26 AI-enabled disinformation campaigns could enable adversaries to propagate false information at an unprecedented scale and seeming authenticity and augment “active measures” capabilities to sow confusion and otherwise influence U.S. society.
Acquisition and Architecture
As rivals such as China and Russia innovate and field emerging technologies into their operations, continuing obstacles to procurement and tech integration will hinder the IC’s ability to keep up or outpace. The Task Force members identified three key issues: acquiring new technologies and infrastructure for AI tools; acquiring the data needed to train and power AI/ML applications; and building an effective data architecture.
Tech Acquisition: The IC’s lengthy research, development, testing, and evaluation timeline reflects its unique risks and security requirements but reduces its agility and speed in acquiring new technologies.27 Procurement and contracting practices hinder the IC’s ability to adapt technologies to legacy systems and restructure key tasks, such as retraining ML algorithms, to shifting intelligence needs and operational environments.28,29 Task Force participants identified the need not only to reform IC acquisition authorities and commercial sector engagement but also for greater risk tolerance from policy and Congressional oversight to allow for experimentation and occasional failure.30 Challenges to tech acquisition do not stem solely from the IC’s end; Task Force participants noted the need for commercial vendors to ensure the speed of government-provided technology services and updates match those provided to commercial clients.
Data Acquisition: In addition to hardware, software, and platforms, the IC also needs data from the private sector and better ability to acquire and access large volumes of commercial datasets to test, train, and power AI/ML algorithms and applications. Acquiring commercial datasets poses two key issues: trust and redundancy. Commercial ML datasets are often globally crowdsourced, creating uncertainty over data integrity that is compounded by vendor opacity over algorithms and models. Rather than acquiring commercial data, IC organizations are exploring providing commercial firms with unclassified government data to combine with open-source datasets, creating a “digital twin” for commercial firms to hone algorithms and datasets to be shared with the IC. The “digital twin” approach, however, is uncertain to generate new, value-added insights.
The IC’s lengthy research, development, testing, and evaluation timeline reflects its unique risks and security requirements but reduces its agility and speed in acquiring new technologies.
Data Architecture and Integration: Along with acquiring quality datasets, the IC must also improve the speed and ease with which data is transferred, stored, and shared on IC architecture. Primarily closed data architecture hampers IC integration with commercial providers, lacking known, open, and unclassified interfaces to effectively integrate data and software. Once on IC architecture, technical barriers still remain in migrating “low-side” unclassified data onto “high-side” classified systems and integrating it into intelligence workflows. Even if data can be merged onto “high-side” data lakes, vertical stovepipes and varying data governance, labeling, and access standards across agencies still hinder AI/ML users from accessing needed data.
Workforce and Workflows
Even as new technologies are acquired, their adoption and assimilation may not align with long-held mission sets, institutional norms, and legacy processes. Enduring tradecraft will need to blend with entirely new skillsets, team structures, intelligence tasks, and performance goals to facilitate deep and long-term technology integration.
Skillsets: Leveraging and integrating AI and other technologies into day-to-day collection and processing tasks and workflows will require a significant rethinking and retooling of the skillsets of collectors and analysts executing those missions in the field and headquarters. Future generations of HUMINT officers will need to arrive with or acquire significant science and technology skills to operate effectively in digital collection and agent handling environments.31 Effective analysts at technical collection agencies such as NGA and NSA will need to continue mastering highly specialized skills such as signals, imagery, and geospatial analysis while gaining literacy and baseline skills in AI/ML and analytics tools to integrate into their analysis.32,33
Teams and Cultures: Technology adoption may change the tools available to collectors and analysts but not necessarily their organizational cultures and leadership attitudes on what skills and missions should be prioritized and valued. In the HUMINT world, one expert noted that a “romanticism” surrounds traditional field tradecraft and “institutional resistance” to change still impact how case officers are recruited, trained, and rewarded, hindering the speed and depth to which digital skills are inculcated into officers and operations.34 Technical collection agency teams remain largely centered on the specific skills and tasks that have defined them for decades, relegating new disciplines such as data scientists and ML engineers to second-tier roles.35
Authenticity: Analysts must continue to evaluate data and intelligence for quality, accuracy, and utility, but with the adoption of AI/ML tools, they also must learn how to measure, judge, and factor in entirely new attributes such as data authenticity into their analysis. Advances in adversarial AI will upend previous assumptions that collected intelligence—albeit poor, inaccurate, or deceiving—was at least real. Ensuring the authenticity of collected imagery, electronic signals, and communications will be critical to usability but grow harder as adversaries grow adept at altering data. HUMINT officers must be able to authenticate both the intelligence collected and the intelligence source if recruited and handled digitally.
Explainability: Along with authenticity, AI integration introduces another key factor analysts must learn how to factor and measure: explainability. As AI/ML tools are incorporated into how products are generated, users may demand to know the logic, assumptions, and data biases of the algorithms used to generate them—which may or may not be knowable. Lack of transparency on evidence chains and how, where, and when AI was applied will leave key users—analysts, operators, and policymakers—uncertain in trusting the resulting products.
Technology adoption may change the tools available to collectors and analysts but not necessarily their organizational cultures and leadership attitudes on what skills and missions should be prioritized and valued.
Implications: Adapting to Evolving Threat—and Collection—Environments
Emerging technologies hold tremendous potential to augment, accelerate, and improve the way intelligence is collected and processed to better serve U.S. national security objectives. While the number of new technologies with collection and processing applications continues to grow, the IC’s capacity to acquire, integrate, and harness them must keep pace but faces two realities. First, unlike previous eras of defense and intel innovation, the United States no longer dominates the global market in creating these technologies, and the IC cannot assume that leading U.S. innovation firms will support the IC mission. Second, rapidly evolving threat environments mean that even the best technology must be adaptable to dramatic shifts in national security priorities.
Indeed, in concluding phase one of this Task Force, it is the need for adaptability in intelligence collection—in where, how, and what is collected—that emerged a central and perhaps organizing principle for the IC in the years ahead. The IC must be adaptable not only in how it employs these tools and technologies but also in how it conceptualizes and executes its missions. This will involve adaptability in, among other areas, how it partners with foreign allies, leverages open-source, and approaches the next generation of disruptive technologies.
Leveraging Commercial Partners: High-end intelligence collection is no longer the sole domain of the U.S. IC and foreign intelligence rivals, as emerging technologies transform intelligence capabilities in the private sector. The commercialization of space and proliferation of satellite-based imaging and sensors will enable the commercial sector to collect quality GEOINT and SIGINT that, when combined with advanced analytics and OSINT data, can generate quality and timely all-source intelligence products.36,37,38 The IC could leverage the commercial sector not only for acquiring technology but also for collaboration or even outsourcing of collection, processing, and baseline analytic tasks while focusing the more “exquisite” IC platforms on harder and priority targets.
Leveraging Allies: As the U.S. IC adopts the best tools and technologies from the commercial sector, it must also deepen and expand its relationships with allied foreign intelligence services at the cutting edge of integrating these technologies into intelligence tradecraft. Beyond traditional intelligence sharing and liaison partnerships, the U.S. IC will need to reconceptualize and adapt how it develops, employs, and exploits emerging technologies with partners at the innovation edge. While cognizant of counterintelligence and security, the capabilities and shared collection priorities of allies and partners (including the Five-Eyes alliance, the North Atlantic Treaty Organization (NATO), Israel, and Asian allies) should be leveraged to enhance the scope, access, and quality of tech-enabled collection and intelligence.
Over-the-Horizon Technologies: While this Task Force is focused on near-term applications of advanced technologies, the study has made clear the vital importance of anticipating the next generation of technologies that will disrupt and transform intelligence collection and the global threat environment. Who becomes the leader in developing, understanding, and integrating these technologies will gain a strategic advantage on the intelligence battlefield. Instead of playing catch-up to technological advances, as the IC arguably has done with AI, the IC must begin shaping a future workforce with the skills, tools, and tradecraft to lead in these fields:
- Biotechnology: As the Covid-19 pandemic has made clear, biology and biotechnology will play a central role in U.S. national security in the coming decades. The IC’s challenge will be not only understanding the nature and implications of pandemics and biological threats but also how adversaries seek to use biotechnologies to advance their strategic interests. Biology, according to Chinese military officials, will be a new domain of warfare—and thus require intelligence to understand and deter the threat. For the IC, biotechnology could alter the very nature of intelligence collection and spawn a new field of bio-intelligence. A convergence of advances in synthetic biology, AI, and computational power could create transformational new collection capabilities in biosensing, biological geolocation, and DNA data storage and transfer. While providing a strategic intelligence advantage, bio-intelligence will also pose profound ethical questions for IC and national security leaders.
- Quantum: Advances in quantum sensing, computing, and networking will impact every means of intelligence collection and the speed, scale, and ways in which data and intelligence are processed. The race for quantum encryption and decryption could determine the future of SIGINT and the IC’s ability to collect and secure intelligence assets, access, and data. Beyond cryptological implications, quantum computing could accelerate and transform all of the AI/ML and cloud computing-based collecting and processing capabilities identified in this brief.
- 5G and IoT: The mass fielding of 5G technology and IoT devices augurs dramatic shifts in where, what, and how intelligence is collected, creating opportunities for collectors but exponentially growing burden for processing. “Ubiquitous connectivity,” disconnection of hardware, and edge cloud computing could enable intelligence to be generated almost anywhere at 5G speed. At the same time, the volume and variety of 5G data and the number of signals and emitters from IoT will make finding useful signals amongst the “digital ocean” exponentially more difficult.
Brian Katz is a fellow in the International Security Program at the Center for Strategic and International Studies (CSIS) and research director of the CSIS Technology and Intelligence Task Force.
This report is made possible by support to the CSIS Technology and Intelligence Task Force from Booz Allen Hamilton, Rebellion Defense, Redhorse, and TRSS.
CSIS Briefs are produced by the Center for Strategic and International Studies (CSIS), a private, tax-exempt institution focusing on international public policy issues. Its research is nonpartisan and nonproprietary. CSIS does not take specific policy positions. Accordingly, all views, positions, and conclusions expressed in this publication should be understood to be solely those of the author(s).
© 2020 by the Center for Strategic and International Studies. All rights reserved.
Please consult the PDF for references.