Publications

Finding the most likely path to a set of failure states is important to the analysis of safety critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates. As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.

The capability to avoid other air traffic is a fundamental component of the layered conflict management system to ensure safe and efficient operations in the National Airspace System. The evaluation of systems designed to mitigate the risk of midair collisions of manned aircraft are based on large-scale modeling and simulation efforts and a quantitative volume defined as a near midair collision (NMAC). Since midair collisions are difficult to observe in simulation and are inherently rare events, basing evaluations on NMAC enables a more robust statistical analysis. However, an NMAC and its underlying assumptions for assessing close encounters with manned aircraft do not adequately consider the different characteristics of smaller UAS-only encounters. The primary contribution of this paper is to explore quantitative criteria to use when simulating two or more smaller UASs in sufficiently close proximity that a midair collision might reasonably occur and without any mitigations to reduce the likelihood of a midair collision. The criteria assumes a historically motivated upper bound for the collision likelihood and subsequently identify the smallest possible NMAC analogs. We also demonstrate the NMAC analogs can be used to support modeling and simulation activities.

Interval Management (IM) is an FAA Next-Gen Automatic Dependent Surveillance – Broadcast (ADS-B) In application designed to decrease the variability in spacing between aircraft, thereby increasing the efficiency of the National Airspace System (NAS). One application within IM is Paired Approach (PA). In a PA operation, the lead aircraft and trail aircraft are both established on final approach to dependent parallel runways with runway centerline spacing less than 2500 feet. The trail aircraft follows speed guidance from the IM Avionics to achieve and maintain a desired spacing behind the lead aircraft. PA operations are expected to require a new separation standard that allows the aircraft to be spaced more closely than current dependent parallel separation standards. The behavior of an airborne collision avoidance system, such as TCAS II or ACAS Xa, must be considered during a new operation such as PA, because the aircraft are so closely spaced. This analysis quantified TAs and RAs using TCAS II Change 7.1 and ACAS Xa software with simulated IM PA operations. The results show no RAs using either TCAS II Change 7.1 or ACAS Xa, negligible TAs using TCAS II Change 7.1, and acceptable numbers of TAs using ACAS Xa software during simulated PA operations.

Small Unmanned Aircraft Systems (sUAS) offer many potential benefits to society but also pose a dangerous mid-air collision hazard. Safely integrating into shared airspace will require sUAS to perform Collision Avoidance (CA), one of the primary components of Detect and Avoid (DAA) technologies. This paper performs a Monte Carlo simulation of close encounters between sUAS and manned aircraft to evaluate the safety and alerting rates of three CA system architecture options: manned aircraft avoiding sUAS, sUAS avoiding manned aircraft, and both types of aircraft avoiding each other. Novel CA policies based on ACAS X are introduced for sUAS. These policies enable sUAS to perform escape maneuvers with far lower vertical climb capabilities than what is expected by current CA systems.

Unmanned aircraft systems (UAS) have proven to have distinct advantages compared to manned aircraft for a variety of tasks. Current airspace regulations require a capability to sense and avoid other aircraft to replace the ability of a pilot to see and avoid other traffic. A prototype phased-array radar was developed and tested to demonstrate a capability to support the sense and avoid (SAA) requirement and to validate radar performance models. Validated radar models enable evaluation of other radar systems in simulation. This paper provides an overview of the unique radar technology, and focuses on radar performance and model validation as demonstrated through a flight testing campaign. Performance results demonstrate that the prototype SAA radar system can provide sufficient accuracy to sense avoid non-cooperative aircraft.