NEXT-GENERATION TECHNOLOGY FOR THE EXTREMELY EFFICIENT STORAGE, DISTRIBUTION, AND PROCESSING OF NUCLEAR PHYSICS DATA -- SBIR Phase I Final Report
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Presentation to be presented during the Site-Directed Research and Development (SDRD) program FY 2021 review meeting (Webex), September 22–23, 2021.
This is a report for the FY2021 annual program overview.
Systems, methods, techniques and apparatuses of asynchronous communication is distributed systems are disclosed. One exemplary embodiment is a method determining, with a plurality of agent nodes structured to communicate asynchronously in a distributed system, a first set of iterations including an iteration determined by each of the plurality of agent nodes; determining, with a first agent node of the plurality of agent nodes, a local vector clock; receiving, with the first agent node, a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration; updating, with the first agent node, the local vector clock based on the received remote vector clock; and determining a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock.
The MapReduce programming model, due to its simplicity and scalability, has become an essential tool for processing large data volumes in distributed environments. Recent Stream Processing Systems (SPS) extend this model to provide low-latency analysis of high-velocity continuous data streams. However, integrating MapReduce with streaming poses challenges: first, the runtime variations in data characteristics such as data-rates and key-distribution cause resource overload, that inturn leads to fluctuations in the Quality of the Service (QoS); and second, the stateful reducers, whose state depends on the complete tuple history, necessitates efficient fault-recovery mechanisms to maintain the desired QoS in the presence of resource failures. We propose an integrated streaming MapReduce architecture leveraging the concept of consistent hashing to support runtime elasticity along with locality-aware data and state replication to provide efficient load-balancing with low-overhead fault-tolerance and parallel fault-recovery from multiple simultaneous failures. Our evaluation on a private cloud shows up to 2:8 improvement in peak throughput compared to Apache Storm SPS, and a low recovery latency of 700 -1500 ms from multiple failures.
We present the NNDC-BNL methodology for estimating neutron cross section covariances in thermal, resolved resonance, unresolved resonance and fast neutron regions. The three key elements of the methodology are Atlas of Neutron Resonances, nuclear reaction code EMPIRE, and the Bayesian code implementing Kalman filter concept. The covariance data processing, visualization and distribution capabilities are integral components of the NNDC methodology. We illustrate its application on examples including relatively detailed evaluation of covariances for two individual nuclei and massive production of simple covariance estimates for 307 materials. Certain peculiarities regarding evaluation of covariances for resolved resonances and the consistency between resonance parameter uncertainties and thermal cross section uncertainties are also discussed.
The ability of recurrent neural networks (RNN) to model nonlinear dynamics of high dimensional process data has enabled data-driven RNN-based fault detection algorithms. Previous studies have focused on detecting faults by identifying the discrepancies in data distribution between the faulty and normal data, as reflected in prediction errors generated by RNN models. However, in industrial processes, variations in data distribution can also result from changes in normal control setpoints and compensatory control adjustments in response to disturbances, making it hard to differentiate between normal and faulty conditions. This paper proposes a fault detection method utilizing a long short-term memory (LSTM) encoder–decoder structure with regularized hidden dynamics and reversible instance normalization (RevIN) to compactly represent high-dimensional measurements for effective monitoring. During training, the hidden states of the model are regularized to form a low-dimensional latent space representation of the original multivariate time series data. As a result, the prediction errors of the latent states can be used to monitor the abnormal dynamic variations, while the reconstruction errors of the measured variables are used to monitor the abnormal static variations. Furthermore, the proposed indices can reflect operating conditions, even when the distribution of test data changes, which helps distinguish faults from normal adjustments and disturbances that controllers can settle. Here, data from numerical simulation and the Tennessee Eastman process are used to illustrate the effectiveness of the proposed fault detection method.
Reduced Instruction Set Computer (RISC) workstations and Personnel Computers (PC) are very popular tools for office automation, command and control, scientific analysis, database management, and many other applications. However, when using Input/Output (I/O) intensive applications, the RISC workstations and PC's are often overburdened with the tasks of collecting, staging, storing, and distributing data. Also, by using standard high-performance peripherals and storage devices, the I/O function can still be a common bottleneck process. Therefore, the high-performance mass storage system, developed by Loral AeroSys' Independent Research and Development (IR&D) engineers, can offload a RISC workstation of I/O related functions and provide high-performance I/O functions and external interfaces. The high-performance mass storage system has the capabilities to ingest high-speed real-time data, perform signal or image processing, and stage, archive, and distribute the data. This mass storage system uses a hierarchical storage structure, thus reducing the total data storage cost, while maintaining high-I/O performance. The high-performance mass storage system is a network of low-cost parallel processors and storage devices. The nodes in the network have special I/O functions such as: SCSI controller, Ethernet controller, gateway controller, RS232 controller, IEEE488 controller, and digital/analog converter. The nodes are interconnected through high-speed direct memory access links to form a network. The topology of the network is easily reconfigurable to maximize system throughput for various applications. This high-performance mass storage system takes advantage of a 'busless' architecture for maximum expandability. The mass storage system consists of magnetic disks, a WORM optical disk jukebox, and an 8mm helical scan tape to form a hierarchical storage structure. Commonly used files are kept in the magnetic disk for fast retrieval. The optical disks are used as archive media, and the tapes are used as backup media. The storage system is managed by the IEEE mass storage reference model-based UniTree software package. UniTree software will keep track of all files in the system, will automatically migrate the lesser used files to archive media, and will stage the files when needed by the system. The user can access the files without knowledge of their physical location. The high-performance mass storage system developed by Loral AeroSys will significantly boost the system I/O performance and reduce the overall data storage cost. This storage system provides a highly flexible and cost-effective architecture for a variety of applications (e.g., realtime data acquisition with a signal and image processing requirement, long-term data archiving and distribution, and image analysis and enhancement).
Uranium particles within the respirable size range pose a significant hazard to the health and safety of workers. Significant differences in the deposition and incorporation patterns of aerosols within the respirable range can be identified and integrated into sophisticated health physics models. Data characterizing the uranium particle size distribution resulting from specific foundry-related processes are needed. Using personal air sampling cascade impactors, particles collected from several foundry processes were sorted by activity median aerodynamic diameter onto various Marple substrates. After an initial gravimetric assessment of each impactor stage, the substrates were analyzed by alpha spectrometry to determine the uranium content of each stage. Alpha spectrometry provides rapid nondestructive isotopic data that can distinguish process uranium from natural sources and the degree of uranium contribution to the total accumulated particle load. In addition, the particle size bins utilized by the impactors provide adequate resolution to determine if a process particle size distribution is: lognormal, bimodal, or trimodal. Data on process uranium particle size values and distributions facilitate the development of more sophisticated and accurate models for internal dosimetry, resulting in an improved understanding of foundry worker health and safety.
This study investigates the sensitivity of moist processes and surface precipitation during three extreme precipitation events over Belgium to the representation of rain, snow and hail size distributions in a bulk one-moment microphysics parameterisation scheme. Sensitivities included the use of empirically derived relations to calculate the slope parameter and diagnose the intercept parameter of the exponential snow and rain size distributions and sensitivities to the treatment of hail/graupel. A detailed evaluation of the experiments against various high temporal resolution and spatially distributed observational data was performed to understand how moist processes responded to the implemented size distribution modifications. Net vapor consumption by microphysical processes was found to be unaffected by snow or rain size distribution modifications, while it was reduced replacing formulations for hail by those typical for graupel, mainly due to intense sublimation of graupel. Cloud optical thickness was overestimated in all experiments and all cases, likely due to overestimated snow amounts. The overestimation slightly deteriorated by modifying the rain and snow size distributions due to increased snow depositional growth, while it was reduced by including graupel. The latter was mainly due to enhanced cloud water collection by graupel and reduced snow depositional growth. Radar reflectivity and cloud optical thickness could only be realistically represented by inclusion of graupel during a stratiform case, while hail was found indispensable to simulate the vertical reflectivity profile and the surface precipitation structure. Precipitation amount was not much altered by any of the modifications made and the general overestimation was only decreased slightly during a supercell convective case.
Abstract Machine learning (ML) models often fail with data that deviates from their training distribution. This is a significant concern for ML-enabled devices as data drift may lead to unexpected performance. This work introduces a new framework for out of distribution (OOD) detection and data drift monitoring that combines ML and geometric methods with statistical process control (SPC). We investigated different design choices, including methods for extracting feature representations and drift quantification for OOD detection in individual images and as an approach for input data monitoring. We evaluated the framework for both identifying OOD images and demonstrating the ability to detect shifts in data streams over time. We demonstrated a proof-of-concept via the following tasks: 1) differentiating axial vs. non-axial CT images, 2) differentiating CXR vs. other radiographic imaging modalities, and 3) differentiating adult CXR vs. pediatric CXR. For the identification of individual OOD images, our framework achieved high sensitivity in detecting OOD inputs: 0.980 in CT, 0.984 in CXR, and 0.854 in pediatric CXR. Our framework is also adept at monitoring data streams and identifying the time a drift occurred. In our simulations tracking drift over time, it effectively detected a shift from CXR to non-CXR instantly, a transition from axial to non-axial CT within few days, and a drift from adult to pediatric CXRs within a day—all while maintaining a low false positive rate. Through additional experiments, we demonstrate the framework is modality-agnostic and independent from the underlying model structure, making it highly customizable for specific applications and broadly applicable across different imaging modalities and deployed ML models.
Topics addressed include: software engineering; university grants; institutes; concurrent processing; sparse distributed memory; distributed operating systems; intelligent data management processes; expert system for image analysis; fault tolerant software; and architecture research.
Geospatial resource assessments frequently require timely geospatial data processing that involves large multivariate remote sensing data sets. In particular, for disasters, response requires rapid access to large data volumes, substantial storage space and high performance processing capability. The processing and distribution of this data into usable information products requires a processing pipeline that can efficiently manage the required storage, computing utilities, and data handling requirements. In recent years, with the availability of cloud computing technology, cloud processing platforms have made available a powerful new computing infrastructure resource that can meet this need. To assess the utility of this resource, this project investigates cloud computing platforms for bulk, automated geoprocessing capabilities with respect to data handling and application development requirements. This presentation is of work being conducted by Applied Sciences Program Office at NASA-Stennis Space Center. A prototypical set of image manipulation and transformation processes that incorporate sample Unmanned Airborne System data were developed to create value-added products and tested for implementation on the "cloud". This project outlines the steps involved in creating and testing of open source software developed process code on a local prototype platform, and then transitioning this code with associated environment requirements into an analogous, but memory and processor enhanced cloud platform. A data processing cloud was used to store both standard digital camera panchromatic and multi-band image data, which were subsequently subjected to standard image processing functions such as NDVI (Normalized Difference Vegetation Index), NDMI (Normalized Difference Moisture Index), band stacking, reprojection, and other similar type data processes. Cloud infrastructure service providers were evaluated by taking these locally tested processing functions, and then applying them to a given cloud-enabled infrastructure to assesses and compare environment setup options and enabled technologies. This project reviews findings that were observed when cloud platforms were evaluated for bulk geoprocessing capabilities based on data handling and application development requirements.
An automated technique for measuring particle diameters and their spatial coordinates from holographic reconstructions is being developed. Preliminary tests on actual cold-flow holograms of impinging jets indicate that a suitable discriminant algorithm consists of a Fourier-Gaussian noise filter and a contour thresholding technique. This process identifies circular as well as noncircular objects. The desired objects (in this case, circular or possibly ellipsoidal) are then selected automatically from the above set and stored with their parametric representations. From this data, dropsize distributions as a function of spatial coordinates can be generated and combustion effects due to hardware and/or physical variables studied.
For over the last 15 years, NASA's Earth Science Enterprise (ESE) has devoted a tremendous effort to design and build the Earth Observing System (EOS) Data and Information System (EOSDIS) to acquire, process, archive and distribute the data of the EOS series of satellites and other ESE missions and field programs. The development of EOSDIS began with an early prototype to support NASA data from heritage missions and progressed through a formal development process to today's system that supports the data from multiple missions including Landsat 7, Terra, Aqua, SORCE and ICESat. The system is deployed at multiple Distributed Active Archive Centers (DAACs) and its current holdings are approximately 4.5 petabytes. The current set of unique users requesting EOS data and information products exceeds 2 million. While EOSDIS has been the centerpiece of NASA's Earth Science Data Systems, other initiatives have augmented the services of EOSDIS and have impacted its evolution and the future directions of data systems within the ESE. ESDIS had an active prototyping effort and has continued to be involved in the activities of the Earth Science Technology Office (ESTO). In response to concerns from the science community that EOSDIS was too large and monolithic, the ESE initiated the Earth Science Information Partners (ESP) Federation Experiment that funded a series of projects to develop specialized products and services to support Earth science research and applications. Last year, the enterprise made 41 awards to successful proposals to the Research, Education and Applications Solutions Network (REASON) Cooperative Agreement Notice to continue and extend the ESP activity. The ESE has also sponsored a formulation activity called the Strategy for the Evolution of ESE Data Systems (SEEDS) to develop approaches and decision support processes for the management of the collection of data system and service providers of the enterprise. Throughout the development of its earth science data systems, NASA has had an active collaboration with a number of interagency and international partners. One of the mechanisms that has been extremely helpful in initiating and promoting this collaboration has been NASA's participation in the Committee on Earth Observation Satellites (CEOS) and its Working Group on Information Systems and Services (WGISS). The CEOS members, working together, have implemented an International Directory Network that enables users to locate collections of earth science data held by the international community and an International Catalog System to search and order specific data products. CEOS WGISS has also promoted the international interest in the Open GIS Consortium s specifications that further advance the access and use of geospatial data and the interoperation of GTS components. These are just a few highlights of the benefits that member agencies gain from CEOS participation.
Visionary concepts of the operational environmental satellite system of the 1990's are presented. These exploit planned and anticipated advances in technology. The system comprises 8 Low Earth Orbitors (LEOS) at 500 km height for global coverage at 3 hourly intervals and 3 Geosynchronous Earth Orbitors (GEOS) for Western Hemisphere coverage with stereo capability and near simultaneous viewing of several regional areas. The LEOS emphasize passive and active microwave and Lidar sensing; the GEOS focus on high resolution visible and IR imaging and passive soundings. The GEOS also serve as data relays from the LEOS and as data collection platforms for hazard-related data. An integrated data management system ties the space fleet together and blends the data with that from all other observing systems to process and distribute the data and products to all users.
The NASA Earth Observation System Data and Information System (EOSDIS) has been operating since 1994 and is serving a global user community with well-managed Earth science data in a variety of scientific disciplines. EOSDIS processes, archives and distributes data and information products resulting from spaceborne and airborne instruments as well as in situ measurements from field campaigns. The Earth Science Data and Information System (ESDIS) Project at the NASA Goddard Space Flight Center manages EOSDIS with its 12 Distributed Active Archive Centers (DAACs) located across the United States. During the entire life of EOSDIS, the ESDIS Project and the DAACs have deployed many different mechanisms for user feedback, which have proven extremely valuable to their evolution and performance in their service to user communities. Some of the inputs from our user groups have resulted in fundamental changes in the architecture, design and operations of EOSDIS, while others have provided novel ideas for incremental changes. The EOSDIS DAACs have User Working Groups (UWGs) that represent broad user communities in the Earth science disciplines served by the DAACs. The UWGs meet periodically to assess and provide feedback on dataset and service priorities. As regular users of the data and services of the DAAC and experts in the scientific disciplines, the UWG members provide valuable inputs for planning and prioritizing the services, as well as addition of new datasets, for the benefit of the community. The EOSDIS is evaluated annually through an independently administered survey of its users resulting in the American Customer Satisfaction Index (ACSI). The survey provides an ACSI score as well as free-text suggestions from users, which are also helpful in making specific system improvements. In addition, each of the DAACs has a user services group that address on-going requests for help and other comments from users. The ESDIS Project has established a mechanism through the "earthdata" website (http://earthdata.nasa.gov) for users to provide feedback of any kind and these questions/comments are routed to the appropriate individuals in the Project or the DAACs. These various forms of receiving user feedback and responding to them continue to be extremely valuable in evolving and sustaining our Earth Science repository.
Purpose: To evaluate the ability of statistical process control methods to detect systematic errors when using a two dimensional (2D) detector array for routine electron beam energy verification. Methods: Electron beam energy constancy was measured using an aluminum wedge and a 2D diode array on four linear accelerators. Process control limits were established. Measurements were recorded in control charts and compared with both calculated process control limits and TG-142 recommended specification limits. The data was tested for normality, process capability and process acceptability. Additional measurements were recorded while systematic errors were intentionally introduced. Systematic errors included shifts in the alignment of the wedge, incorrect orientation of the wedge, and incorrect array calibration. Results: Control limits calculated for each beam were smaller than the recommended specification limits. Process capability and process acceptability ratios were greater than one in all cases. All data was normally distributed. Shifts in the alignment of the wedge were most apparent for low energies. The smallest shift (0.5 mm) was detectable using process control limits in some cases, while the largest shift (2 mm) was detectable using specification limits in only one case. The wedge orientation tested did not affect the measurements as this did not affect the thickness of aluminum over the detectors of interest. Array calibration dependence varied with energy and selected array calibration. 6 MeV was the least sensitive to array calibration selection while 16 MeV was the most sensitive. Conclusion: Statistical process control methods demonstrated that the data distribution was normally distributed, the process was capable of meeting specifications, and that the process was centered within the specification limits. Though not all systematic errors were distinguishable from random errors, process control limits increased the ability to detect systematic errors using routine measurement of electron beam energy constancy.