Publications
Adaptive stress testing: finding likely failure events with reinforcement learning
December 1, 2020
Journal Article
Published in:
J. Artif. Intell. Res., Vol. 69, 2020, pp. 1165-1201.
Summary
Finding the most likely path to a set of failure states is important to the analysis of safety critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates. As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.
Summary
Finding the most likely path to a set of failure states is important to the analysis of safety critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due...
READ MORE
A quantitatively derived NMAC analog for smaller unmanned aircraft systems based on unmitigated collision risk
November 19, 2020
Journal Article
Published in:
Preprints, 19 November 2020.
Summary
The capability to avoid other air traffic is a fundamental component of the layered conflict management system to ensure safe and efficient operations in the National Airspace System. The evaluation of systems designed to mitigate the risk of midair collisions of manned aircraft are based on large-scale modeling and simulation efforts and a quantitative volume defined as a near midair collision (NMAC). Since midair collisions are difficult to observe in simulation and are inherently rare events, basing evaluations on NMAC enables a more robust statistical analysis. However, an NMAC and its underlying assumptions for assessing close encounters with manned aircraft do not adequately consider the different characteristics of smaller UAS-only encounters. The primary contribution of this paper is to explore quantitative criteria to use when simulating two or more smaller UASs in sufficiently close proximity that a midair collision might reasonably occur and without any mitigations to reduce the likelihood of a midair collision. The criteria assumes a historically motivated upper bound for the collision likelihood and subsequently identify the smallest possible NMAC analogs. We also demonstrate the NMAC analogs can be used to support modeling and simulation activities.
Summary
The capability to avoid other air traffic is a fundamental component of the layered conflict management system to ensure safe and efficient operations in the National Airspace System. The evaluation of systems designed to mitigate the risk of midair collisions of manned aircraft are based on large-scale modeling and simulation...
READ MORE
Frequency of ADS-B equipped manned aircraft observed by the OpenSky Network
November 12, 2020
Conference Paper
Published in:
8th OpenSky Symp. 2020, Online, 12–13 November 2020.
Summary
To support integration of unmanned aerial systems into the airspace, the low altitude airspace needs to be characterized. Identifying the frequency of different aircraft types, such as rotorcraft or fixed wing single engine, given criteria such as altitude, airspace class, or quantity of seats can inform surveillance requirements, flight test campaigns, or simulation safety thresholds for detect and avoid systems. We leveraged observations of Automatic Dependent Surveillance-Broadcast (ADS-B) equipped aircraft by the OpenSky Network for this characterization.
Summary
To support integration of unmanned aerial systems into the airspace, the low altitude airspace needs to be characterized. Identifying the frequency of different aircraft types, such as rotorcraft or fixed wing single engine, given criteria such as altitude, airspace class, or quantity of seats can inform surveillance requirements, flight test...
READ MORE
Multi-Agent Systems Collaborative Teaming (MASCOT) definition process to create specifications for Multi-Agent System (MAS) development
November 2, 2020
Conference Paper
Published in:
25th Intl. Command and Control Research and Technology Symp., ICCRTS 2020, 2-5 November 2020.
Topic:
R&D area:
R&D group:
Summary
The US Army envisions heterogeneous teams of advanced machines and humans that will collaborate together to achieve a common mission goal. It is essential for commanders to quickly and effectively respond to dynamic mission environments with agile re-tasking and computerized aids for plan definition/redefinition, and to perform some tasks with bounded autonomy. Workload constraints limit an individual's ability to concurrently control many platforms, so some mission segments many need to be autonomous or to be quickly selected via a tactics playbook. Denied environments also dictate the need for machine participants in some mission segments to be autonomous (or semi-autonomous). A Multi-Agent System (MAS) provides a natural paradigm for describing a system of agents that work together in such environments. An agent can be a human or machine, but is generally a machine. Creating MAS systems and requirements has proved to be a formidable task due to mission complexities, the necessity to deal with unforeseen circumstances, and the general difficulty of defining autonomous behaviors. We define a process called Multi-Agent Systems Collaborative Teaming (MASCOT) Definition Process that starts with a Subject Matter Experts (SME), produces a set of agent specifications, and derives system requirements in sufficient detail to define a MAS that can be modeled in a test-bed, used for facilitation of a safety analysis, and produced into an actual system. The MASCOT process also enables concurrent development of an effects based ontology. We demonstrate the MASCOT process on an example case study to show the efficacy of our process.
Summary
The US Army envisions heterogeneous teams of advanced machines and humans that will collaborate together to achieve a common mission goal. It is essential for commanders to quickly and effectively respond to dynamic mission environments with agile re-tasking and computerized aids for plan definition/redefinition, and to perform some tasks with...
READ MORE
Method to characterize potential UAS encounters using open source data
November 1, 2020
Journal Article
Published in:
Aerospace, Vol. 7, No. 11, November 2020, art. no. 158.
Summary
As unmanned aerial systems (UASs) increasingly integrate into the US national airspace system, there is an increasing need to characterize how commercial and recreational UASs may encounter each other. To inform the development and evaluation of safety critical technologies, we demonstrate a methodology to analytically calculate all potential relative geometries between different UAS operations performing inspection missions. This method is based on a previously demonstrated technique that leverages open source geospatial information to generate representative unmanned aircraft trajectories. Using open source data and parallel processing techniques,we performed trillions of calculations to estimate the relative horizontal distance between geospatial points across sixteen locations.
Summary
As unmanned aerial systems (UASs) increasingly integrate into the US national airspace system, there is an increasing need to characterize how commercial and recreational UASs may encounter each other. To inform the development and evaluation of safety critical technologies, we demonstrate a methodology to analytically calculate all potential relative geometries...
READ MORE
TCAS II and ACAS Xa traffic and resolution advisories during interval management paired approach operations
October 11, 2020
Conference Paper
Published in:
2020 AIAA/IEEE 39th Digital Avionics Systems Conf., DASC, 11-15 October 2020.
Topic:
R&D area:
R&D group:
Summary
Interval Management (IM) is an FAA Next-Gen Automatic Dependent Surveillance – Broadcast (ADS-B) In application designed to decrease the variability in spacing between aircraft, thereby increasing the efficiency of the National Airspace System (NAS). One application within IM is Paired Approach (PA). In a PA operation, the lead aircraft and trail aircraft are both established on final approach to dependent parallel runways with runway centerline spacing less than 2500 feet. The trail aircraft follows speed guidance from the IM Avionics to achieve and maintain a desired spacing behind the lead aircraft. PA operations are expected to require a new separation standard that allows the aircraft to be spaced more closely than current dependent parallel separation standards. The behavior of an airborne collision avoidance system, such as TCAS II or ACAS Xa, must be considered during a new operation such as PA, because the aircraft are so closely spaced. This analysis quantified TAs and RAs using TCAS II Change 7.1 and ACAS Xa software with simulated IM PA operations. The results show no RAs using either TCAS II Change 7.1 or ACAS Xa, negligible TAs using TCAS II Change 7.1, and acceptable numbers of TAs using ACAS Xa software during simulated PA operations.
Summary
Interval Management (IM) is an FAA Next-Gen Automatic Dependent Surveillance – Broadcast (ADS-B) In application designed to decrease the variability in spacing between aircraft, thereby increasing the efficiency of the National Airspace System (NAS). One application within IM is Paired Approach (PA). In a PA operation, the lead aircraft and...
READ MORE
Enhanced parallel simulation for ACAS X development
September 22, 2020
Conference Paper
Published in:
2020 IEEE High Performance Extreme Computing Conf., HPEC, 22-24 September 2020.
Topic:
R&D area:
R&D group:
Summary
ACAS X is the next generation airborne collision avoidance system intended to meet the demands of the rapidly evolving U.S. National Airspace System (NAS). The collision avoidance safety and operational suitability of the system are optimized and continuously evaluated by simulating billions of characteristic aircraft encounters in a fast-time Monte Carlo environment. There is therefore an inherent computational cost associated with each ACAS X design iteration and parallelization of the simulations is necessary to keep up with rapid design cycles. This work describes an effort to profile and enhance the parallel computing infrastructure deployed on the computing resources offered by the Lincoln Laboratory Supercomputing Center. The approach to large-scale parallelization of our fast-time airspace encounter simulation tool is presented along with corresponding parallel profile data collected on different kinds of compute nodes. A simple stochastic model for distributed simulation is also presented to inform optimal work batching for improved simulation efficiency. The paper concludes with a discussion on how this high-performance parallel simulation method enables the rapid safety-critical design of ACAS X in a fast-paced iterative design process.
Summary
ACAS X is the next generation airborne collision avoidance system intended to meet the demands of the rapidly evolving U.S. National Airspace System (NAS). The collision avoidance safety and operational suitability of the system are optimized and continuously evaluated by simulating billions of characteristic aircraft encounters in a fast-time Monte...
READ MORE
Processing of crowdsourced observations of aircraft in a high performance computing environment
September 22, 2020
Conference Paper
Published in:
2020 IEEE High Performance Extreme Computing Conf., HPEC, 22-24 September 2020.
Summary
As unmanned aircraft systems (UASs) continue to integrate into the U.S. National Airspace System (NAS), there is a need to quantify the risk of airborne collisions between unmanned and manned aircraft to support regulation and standards development. Both regulators and standards developing organizations have made extensive use of Monte Carlo collision risk analysis simulations using probabilistic models of aircraft flight. We've previously determined that the observations of manned aircraft by the OpenSky Network, a community network of ground-based sensors, are appropriate to develop models of the low altitude environment. This works overviews the high performance computing workflow designed and deployed on the Lincoln Laboratory Supercomputing Center to process 3.9 billion observations of aircraft. We then trained the aircraft models using more than 250,000 flight hours at 5,000 feet above ground level or below. A key feature of the workflow is that all the aircraft observations and supporting datasets are available as open source technologies or been released to the public domain.
Summary
As unmanned aircraft systems (UASs) continue to integrate into the U.S. National Airspace System (NAS), there is a need to quantify the risk of airborne collisions between unmanned and manned aircraft to support regulation and standards development. Both regulators and standards developing organizations have made extensive use of Monte Carlo...
READ MORE
Detect-and-avoid closed-loop evaluation of noncooperative well clear definitions
July 12, 2020
Journal Article
Published in:
J. Air Transp., Vol. 28, No. 4, 12 July 2020, pp. 195-206.
Topic:
R&D area:
R&D group:
Summary
Four candidate detect-and-avoid well clear definitions for unmanned aircraft systems encountering noncooperative aircraft are evaluated using safety and operational suitability metrics. These candidates were proposed in previous research based on unmitigated collision risk, maneuver initiation ranges, and other considerations. Noncooperative aircraft refer to aircraft without a functioning transponder. One million encounters representative of the assumed operational environment for the detect-and-avoid system are simulated using a benchmark alerting and guidance algorithm as well as a pilot response model. Results demonstrate sensitivity of the safety metrics to the unmanned aircraft’s speed and the detect-and-avoid system's surveillance volume. The only candidate without a horizontal time threshold, named modified tau, outperforms the other three candidates in avoiding losses of detect and avoid well clear. Furthermore, this candidate's alerting timeline lowers the required surveillance range. This can help reduce the barrier of enabling unmanned aircraft systems' operations with low size, weight, and power sensors.
Summary
Four candidate detect-and-avoid well clear definitions for unmanned aircraft systems encountering noncooperative aircraft are evaluated using safety and operational suitability metrics. These candidates were proposed in previous research based on unmitigated collision risk, maneuver initiation ranges, and other considerations. Noncooperative aircraft refer to aircraft without a functioning transponder. One million...
READ MORE
Wind information requirements for NextGen applications phase 7 report
February 12, 2020
Project Report
Published in:
MIT Lincoln Laboratory Report ATC-444
Topic:
R&D area:
Summary
This report details the Required Time of Arrival (RTA) performance of B757 aircraft arriving at various meter fixes across a range of altitudes from 33,000' down to 3,000' above ground level (AGL). The system tested demonstrated less than ±10 second arrival error in at least 95% of flights at meter fixes down to 7,000' AGL regardless of the forecast quality provided. Below 7,000' AGL, RTA performance significantly degraded demonstrating around 80% compliance under the best forecast and operating conditions. This report also provides a comprehensive lexicon of aviation and air traffic control related "wind" terms.
Summary
This report details the Required Time of Arrival (RTA) performance of B757 aircraft arriving at various meter fixes across a range of altitudes from 33,000' down to 3,000' above ground level (AGL). The system tested demonstrated less than ±10 second arrival error in at least 95% of flights at meter...
READ MORE