Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Management Platform”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Findings on subtask 3.3 – applicability of automated brine chemistry determinations for treatment and recovery processes through facility automation/modularization: engineering design study

Under the Energy & Environmental Research Center’s (EERC’s) ~$\$$22 million Phase II Brine Extraction and Storage Test (BEST) Program, a multimillion-dollar brine treatment technology test bed facility was established in western North Dakota to provide a platform for evaluating developing technologies and approaches for brine treatment and volume reduction. The initial facility, and associated research effort, was funded by the U.S. Department of Energy (DOE), with in-kind contributions provided by several industry participants and the state of North Dakota. Since its opening, the Brine Technology Test Facility (BTTF) has supported performance evaluations of desalination technologies capable of treating high-salinity produced water (PW) and enabled data collection for multiple approaches of PW management and critical material recovery. As part of the decommissioning process for the original project, facility ownership and liability were transferred to Select Water, which is providing the EERC with a continuing site access option for state or federal research and/or commercial technology development. This report documents the findings from a design study conducted by the EERC and the engineering firm that was originally contracted to design and construct the facility (Advanced Engineering and Environmental Services, LLC [AE2S]) that evaluated the current status of BTTF and its systems and developed a retrofit design to increase the facility’s capabilities through automation and modularization of its PW treatment infrastructure. The proposed retrofit will provide DOE and industry with an expanded range of conditioned PW that can be produced at the facility for evaluating fit-for-purpose water treatment technologies, online instrumentation for brine chemistry determination, and systems that recover critical materials like lithium and magnesium. The current facility consists of an 9600-square-foot facility that includes a 40-foot by 65-foot Class 1, Division 2-rated demonstration area and associated control rooms and lab-ready space capable of sourcing oil and gas PW and wastewater from industrial sources or tailoring brine compositions up to 300,000 mg/L total dissolved solids (TDS) and supplying them at rates up to 25 gpm for extended-duration technology demonstrations. The colocation of the facility with Select Water’s water management facilities allows for access and unloading of more than 10,000 bbl/day of trucked water delivered to site and associated access to on-site Class I and Class II brine disposal wells and nearby hazardous waste landfills operated and/or contracted by Select Water to dispose of concentrate and/or effluents associated with the testing.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS1990

Cost and quality planning for large NASA programs

The Software Cost and Quality Engineering methodology developed over the last two decades at IBM Federal Sector Div. is used to plan the NASA Space Station Data Management System (DMS). An ongoing project to capture this methodology, which is built on a foundation of experiences and lessons learned, has resulted in the development of a PC-based tool that integrates cost and quality forecasting methodologies and data in a consistent manner. This tool, Software Cost and Quality Engineering Starter Set (SCQESS), is being used to assist in the DMS costing exercises. At the same time, DMS planning serves as a forcing function and provides a platform for the continuing, iterative development, calibration, and validation and verification of SCQESS. The data that forms the cost and quality engineering data base is derived from more than 17 years of development of NASA Space Shuttle software, ranging from low criticality, low complexity support tools to highly complex and highly critical onboard software.

Rone, Kyle Y.↗
NASA NTRSDate not supplied

The Future of NASA Earth Science in the Commercial Cloud: Challenges and Opportunities

NASA produces a large volume and variety of data products that are used every day to support research, decision making, and education. The widespread use of NASA’s Earth Science data is enabled by NASA’s Earth Science Data System (ESDS) program, which oversees the archiving and distribution of these data and invests in the development of new data systems and tools. However, NASA’s current approach to Earth Science data distribution — based on distributed institutional archives with individual on-premises high-performance computing capabilities — faces some significant challenges, including massive increases in data volume from upcoming missions, a greater need for transdisciplinary science that synthesizes many different kinds of observations, and a push to make science more open, inclusive, and accessible. To address these challenges, NASA is aggressively migrating its Earth Science data and related tools and services into the commercial cloud. Migration of data into the commercial cloud can significantly improve NASA’s existing data system capabilities by (1) providing more flexible options for storage and compute (including rapid, as-needed access to state-of-the-art capabilities); (2) by centralizing and standardizing data access, which gives all of NASA’s institutional data centers access to all of each other’s datasets; and (3) by facilitating “analysis-in-place”, whereby users can bring their own computational workflows and tools to the data rather than having to maintain their own copies of NASA datasets. However, migration to the commercial cloud also poses some significant challenges, including (1) managing costs under a “pay-as-you-go” model; (2) incompatibility with existing tools and data formats with object-based storage and network access; (3) vendor lock-in; (4) challenges with data access for workflows that mix on-premise and cloud computing; and (5) standardization for highly diverse data as is present in NASA’s data archive. I conclude with two examples of recent NASA activities showcasing capabilities enabled by the commercial cloud: An interactive analysis and development platform for analyzing airborne imaging spectroscopy data, and a new collection of tools and services for data discovery, analysis, publication, and data-driven storytelling (Visualization, Exploration, and Data Analysis, VEDA).

Alexey N Shiklomanov↗
DOE OSTI2016

DOE Voluntary Partnership Program with Utilities and Local Governments Supports the Design of New Data Access Solutions

The Better Buildings Energy Data Accelerator (BBEDA) is a unique effort that has supported 22 pairs of local governments and their utility companies to help building owners gain access to their whole-building energy data. Municipal and Utility BBEDA Partners committed to develop streamlined and easy-to-use solutions to provide whole-building energy data, especially for multitenant commercial buildings, by the end of 2015. As a result, building owners would be able to make data-driven decisions about their buildings by utilizing readily available energy consumption data for entire buildings. Traditionally, data access was difficult to implement due to technical barriers and the lack of clear value propositions for the utilities. During the past two years, BBEDA has taken a hands-on approach to overcome these barriers by offering a platform for the partners to discuss their challenges and solutions. Customized support was also provided to Partners building their local strategies. Based on the lessons learned from the partners, BBEDA developed a final toolkit with guiding documents that addressed key barriers and served as a resource for the other cities and utilities attempting to establish whole-building data access, including an exploration of opportunities to apply the whole-building data to various aspects of utility demand-side management (DSM) programs. BBEDA has been a catalyst for market transformation by addressing the upstream (to efficiency implementation) barrier of data access, demonstrated through the success of the BBEDA partners to address policy, engagement, and technical hurdles and arrive at replicable solutions to make data access a standard practice nationwide. As a result of best practices identified by the BBEDA, 18 utilities serving more than 2.6 million commercial customers nationwide will provide whole-building energy data access to building owners by 2017. This historic expansion of data accessibility will increase building energy benchmarking, the first step many building owners take to improve the energy efficiency of their buildings.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
NASA NTRS2016

Mobile Launch Platform Vehicle Assembly Area (SWMU 056) Biosparge Expansion Interim Measures Work Plan

This document presents the design details for an Interim Measure (IM) Work Plan (IMWP) for the Mobile Launch Platform/Vehicle Assembly Building (MLPV) Area, located at the John F. Kennedy Space Center (KSC), Florida. The MLPV Area has been designated Solid Waste Management Unit Number 056 (SWMU 056) under KSC's Resource Conservation and Recovery Act (RCRA) Corrective Action Program. This report was prepared by Geosyntec Consultants (Geosyntec) for the National Aeronautics and Space Administration (NASA) under contract number NNK09CA02B and NNK12CA13B, project control number ENV1642. The Advanced Data Package (ADP) presentation covering the elements of this IMWP report received KSC Remediation Team (KSCRT) approval at the December 2015 Team Meeting; the meeting minutes are included in Appendix A.

IMWP↗
DOE OSTI2021

Monitoring agroecosystem productivity and phenology at a national scale: A metric assessment framework

Effective measurement of seasonal variations in the timing and amount of production is critical to managing spatially heterogeneous agroecosystems in a changing climate. Although numerous technologies for such measurements are available, their relationships to one another at a continental extent are unknown. Using data collected from across the Long-Term Agroecosystem Research (LTAR) network and other networks, we investigated correlations among key metrics representing primary production, phenology, and carbon fluxes in croplands, grazing lands, and crop-grazing integrated systems across the continental U.S. Metrics we examined included gross primary productivity (GPP) estimated from eddy covariance (EC) towers and modelled from the Landsat satellite, Landsat NDVI, and vegetation greenness (Green Chromatic Coordinate, GCC) from tower-mounted PhenoCams for 2017 and 2018. Overall, our analysis compared production dynamics estimated from three independent ground and remote platforms using data for 34 agricultural sites constituting 51 site-years of co-located time series. Pairwise sensor comparisons across all four metrics revealed stronger correlation and lower root mean square error (RMSE) between end of season (EOS) dates (Pearson R ranged from 0.6 to 0.7 and RMSE from 32.5 to 67.8) than start of season (SOS) dates (0.46 to 0.69 and 40.4 to 66.2). Overall, moderate to high correlations between SOS and EOS metrics complemented one another except at some lower productivity grazing land sites where estimating SOS can be challenging. Growing season length estimates derived from 16-day satellite GPP (179.1 days) were significantly longer than those from PhenoCam G CC (70.4 days, p adj < 0.0001) and EC GPP (79.6 days, p adj < 0.0001). Landscape heterogeneity did not explain differences in SOS and EOS estimates. Annual integrated estimates of productivity from EC GPP and PhenoCam G CC diverged from those estimated by Landsat GPP and NDVI at sites where annual production exceeds 1000 gC/m –2 yr –1 . Based on our results, we developed a “metric assessment framework” that articulates where and how metrics from satellite, eddy covariance and PhenoCams complement, diverge from, or are redundant with one another. The framework was designed to optimize instrumentation selection for monitoring, modeling, and forecasting ecosystem functioning with the ultimate goal of informing decision-making by land managers, policy-makers, and industry leaders working at multiple scales.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Accomplishments of Thermal Neutron Scattering Research at North Carolina State University [Slides]

Thermal neutron scattering law (TSL) data evaluations have been completed and contributed to ENDF. New evaluations are underway. FLASSH is developing as a modern platform for thermal neutron data analysis featuring an enhanced user experience and low learning overhead. Advanced TSL techniques in combination with Doppler analysis have been developed, which were incorporated into FLASSH. ML/DL NeTS method development has been initiated. Experimental capabilities for validation have been developed at the PULSTAR reactor.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2018

Forecasting Responses of a Northern Peatland Carbon Cycle to Elevated CO 2 and a Gradient of Experimental Warming

The ability to forecast ecological carbon cycling is imperative to land management in a world where past carbon fluxes are no longer a clear guide in the Anthropocene. However, carbon–flux forecasting has not been practiced routinely like numerical weather prediction. This study explored (1) the relative contributions of model forcing data and parameters to uncertainty in forecasting flux– versus pool–based carbon cycle variables and (2) the time points when temperature and CO 2 treatments may cause statistically detectable differences in those variables. We developed an online forecasting workflow (Ecological Platform for Assimilation of Data (EcoPAD)), which facilitates iterative data–model integration. EcoPAD automates data transfer from sensor networks, data assimilation, and ecological forecasting. We used the Spruce and Peatland Responses Under Changing Experiments data collected from 2011 to 2014 to constrain the parameters in the Terrestrial Ecosystem Model, forecast carbon cycle responses to elevated CO 2 and a gradient of warming from 2015 to 2024, and specify uncertainties in the model output. Our results showed that data assimilation substantially reduces forecasting uncertainties. Interestingly, we found that the stochasticity of future external forcing contributed more to the uncertainty of forecasting future dynamics of C flux–related variables than model parameters. However, the parameter uncertainty primarily contributes to the uncertainty in forecasting C pool–related response variables. Given the uncertainties in forecasting carbon fluxes and pools, our analysis showed that statistically different responses of fast–turnover pools to various CO 2 and warming treatments were observed sooner than slow–turnover pools. In conclusion, our study has identified the sources of uncertainties in model prediction and thus leads to improve ecological carbon cycling forecasts in the future.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2016

Estimating Biofuel Feedstock Water Footprints Using System Dynamics

Increased biofuel production has prompted concerns about the environmental tradeoffs of biofuels compared to petroleum-based fuels. Biofuel production in general, and feedstock production in particular, is under increased scrutiny. Water footprinting (measuring direct and indirect water use) has been proposed as one measure to evaluate water use in the context of concerns about depleting rural water supplies through activities such as irrigation for large-scale agriculture. Water footprinting literature has often been limited in one or more key aspects: complete assessment across multiple water stocks (e.g., vadose zone, surface, and ground water stocks), geographical resolution of data, consistent representation of many feedstocks, and flexibility to perform scenario analysis. We developed a model called BioSpatial H2O using a system dynamics modeling and database framework. BioSpatial H2O could be used to consistently evaluate the complete water footprints of multiple biomass feedstocks at high geospatial resolutions. BioSpatial H2O has the flexibility to perform simultaneous scenario analysis of current and potential future crops under alternative yield and climate conditions. In this proof-of-concept paper, we modeled corn grain (Zea mays L.) and soybeans (Glycine max) under current conditions as illustrative results. BioSpatial H2O links to a unique database that houses annual spatially explicit climate, soil, and plant physiological data. Parameters from the database are used as inputs to our system dynamics model for estimating annual crop water requirements using daily time steps. Based on our review of the literature, estimated green water footprints are comparable to other modeled results, suggesting that BioSpatial H2O is computationally sound for future scenario analysis. Our modeling framework builds on previous water use analyses to provide a platform for scenario-based assessment. BioSpatial H2O's system dynamics is a flexible and user-friendly interface for on-demand, spatially explicit, water use scenario analysis for many US agricultural crops. Built-in controls permit users to quickly make modifications to the model assumptions, such as those affecting yield, and to see the implications of those results in real time. BioSpatial H2O's dynamic capabilities and adjustable climate data allow for analyses of water use and management scenarios to inform current and potential future bioenergy policies. The model could also be adapted for scenario analysis of alternative climatic conditions and comparison of multiple crops. The results of such an analysis would help identify risks associated with water use competition among feedstocks in certain regions. Results could also inform research and development efforts that seek to reduce water-related risks of biofuel pathways.

09 BIOMASS FUELS↗
NASA NTRS2000

ERAST: Scientific Applications and Technology Commercialization

This is a conference publication for an event designed to inform potential contractors and appropriate personnel in various scientific disciplines that the ERAST (Environmental Research Aircraft and Sensor Technology) vehicles have reached a certain level of maturity and are available to perform a variety of missions ranging from data gathering to telecommunications. There are multiple applications of the technology and a great many potential commercial and governmental markets. As high altitude platforms, the ERAST vehicles can gather data at higher resolution than satellites and can do so continuously, whereas satellites pass over a particular area only once each orbit. Formal addresses are given by Rich Christiansen, (Director of Programs, NASA Aerospace Technology Ent.), Larry Roeder, (Senior Policy Advisor, U.S. Dept. of State), and Dr. Marianne McCarthy, (DFRC Education Dept.). The Commercialization Workshop is chaired by Dale Tietz (President, New Vista International) and the Science Workshop is chaired by Steve Wegener, (Deputy Manager of NASA ERAST, NASA Ames Research Center.

Hunley, John D.↗
NASA NTRS2019

Assessing the Use of SAR/Optical Data Fusion and TensorFlow for Improved Mangrove Mapping

Mangrove forests are found in intertidal zones of tropical regions around the world and provide important ecological and economic benefits – they are considered carbon sequesters, habitats for flora and fauna, and natural barriers to hurricanes and tsunamis. Wood from mangrove forests are used as fuel and building materials in surrounding coastal communities, therefore promoting local livelihoods. Despite the importance of these ecosystems, mangrove forests have historically been degraded in natural processes such as severe weather, and anthropogenic factors like conversion to agriculture and aquaculture. This study assesses change in mangrove forests in Nigeria and Mozambique from 2015 to 2018 using SAR and optical data fusion. Due to frequent cloud cover over the study area, SAR and optical data is fused to obtain gap-free imagery without clouds. Landsat-8 OLI and Sentinel-1 imagery is fused with TensorFlow, an open source platform used in developing machine learning models. The resulting images are classified to discriminate mangrove forest cover from other land cover types, and change is estimated using image differencing. Understanding the rates and magnitude of mangrove change across space and time can aid in identifying priority areas for forest regeneration, and can help construct sustainable management practices for the future.

Strattman, Katherine↗
NASA NTRS1981

Science and applications space platform

The concept for the Platform/Power System combination (supported by Shuttle visits) is presented from the NASA-sponsored Phase A study. The combination will cost-effectively free-fly small groups of unmanned low-earth payloads which have common accommodation needs. A manned support module and power system can provide elemental and growth capability for 3-6 month missions. The configuration and payload accommodations of the combination systems are reviewed and schematized in detail. Flight operations for the payload science management will include the conduct of science operations, the monitoring of instrument performance, verification of the science data and the maintenance of the science mission plan.

Nein, M.↗
DOE OSTI2019

Using simple PID-inspired controllers for online resilient resource management of distributed scientific workflows

Scientific workflows have become mainstream for conducting large-scale scientific research. As a result, many workflow applications and Workflow Management Systems (WMSs) have been developed as part of the cyberinfrastructure to allow scientists to execute their applications seamlessly on a range of distributed platforms. Although the scientific community has addressed this challenge from both theoretical and practical approaches, failure prediction, detection, and recovery still raise many research questions. In this paper, we propose an approach inspired by the control theory developed as part of autonomic computing to predict failures before they happen, and mitigated them when possible. The proposed approach is inspired on the proportional–integral–derivative controller (PID controller) control loop mechanism, which is widely used in industrial control systems, where the controller will react to adjust its output to mitigate faults. PID controllers aim to detect the possibility of a non-steady state far enough in advance so that an action can be performed to prevent it from happening. To demonstrate the feasibility of the approach, we tackle two common execution faults of large scale data-intensive workflows—data storage overload and memory overflow. We developed a simulator, which implements and evaluates simple standalone PID-inspired controllers to autonomously manage data and memory usage of a data-intensive bioinformatics workflow that consumes/produces over 4.4 TB of data, and requires over 24 TB of memory to run all tasks concurrently. Experimental results obtained via simulation indicate that workflow executions may significantly benefit from the controller-inspired approach, in particular under online and unknown conditions. Simulation results show that nearly-optimal executions (slowdown of 1.01) can be attained when utilizing our proposed method, and faults are detected and mitigated far in advance of their occurrence.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2000

Automation of the CFD Process on Distributed Computing Systems

A script system was developed to automate and streamline portions of the CFD process. The system was designed to facilitate the use of CFD flow solvers on supercomputer and workstation platforms within a parametric design event. Integrating solver pre- and postprocessing phases, the fully automated ADTT script system marshalled the required input data, submitted the jobs to available computational resources, and processed the resulting output data. A number of codes were incorporated into the script system, which itself was part of a larger integrated design environment software package. The IDE and scripts were used in a design event involving a wind tunnel test. This experience highlighted the need for efficient data and resource management in all parts of the CFD process. To facilitate the use of CFD methods to perform parametric design studies, the script system was developed using UNIX shell and Perl languages. The goal of the work was to minimize the user interaction required to generate the data necessary to fill a parametric design space. The scripts wrote out the required input files for the user-specified flow solver, transferred all necessary input files to the computational resource, submitted and tracked the jobs using the resource queuing structure, and retrieved and post-processed the resulting dataset. For computational resources that did not run queueing software, the script system established its own simple first-in-first-out queueing structure to manage the workload. A variety of flow solvers were incorporated in the script system, including INS2D, PMARC, TIGER and GASP. Adapting the script system to a new flow solver was made easier through the use of object-oriented programming methods. The script system was incorporated into an ADTT integrated design environment and evaluated as part of a wind tunnel experiment. The system successfully generated the data required to fill the desired parametric design space. This stressed the computational resources required to compute and store the information. The scripts were continually modified to improve the utilization of the computational resources and reduce the likelihood of data loss due to failures. An ad-hoc file server was created to manage the large amount of data being generated as part of the design event. Files were stored and retrieved as needed to create new jobs and analyze the results. Additional information is contained in the original.

Tejnil, Ed↗
DOE OSTI2020

Approach to Quality Assurance for Complex Environmental Modeling - 20407

A strong Quality Assurance (QA) program for complex environmental modeling is essential for regulatory and public acceptance and trust, but it does not have to be onerous. Neptune and Company, Inc. (Neptune) has developed a strong QA program that improves transparency, traceability, reproducibility, and therefore, defensibility and trust. Neptune's QA program has evolved over the past 27 years, transitioning from an ad hoc QA program, to a program that is currently Nuclear Quality Assurance-1 (NQA-1) compliant, and will soon be NQA-1/DOE approved for DOE EM modeling work. In addition, Neptune is a qualified laboratory assessor and qualified auditor for reference materials, a proficiency testing provider, and fully complies with the Environmental Protection Agency (EPA) QA program. Neptune's President and CEO, Kelly Black, was recently named the Chairperson of the International Organization for Standardization (ISO) Technical Committee 69, Application of Statistical Methods. A strong QA program has been developed for Neptune's radiological performance assessment (PA) program. Although pieces of their QA program are currently in development, their current program includes document control using Subversion, issue tracking and work flow tracking using JIRA, and transparency and traceability via a system of 'calc sheets' for documentation of all data analysis and modeling combined with 'check print' documentation of QA checking, and rigorous model testing and configuration control. All work is reviewed by an independent subject matter expert who is not associated with the collection and assembly of information, for an internal peer review. The program is enforced with a handful of Standard Operation Procedures (SOPs), Work Instructions, QA Project Plans (QAPPs), and Quality Management Plans (QMPs) that are updated frequently, with required annual training and acknowledgment. In addition, effective communication of modeling approaches and results to clients and stakeholders are integral to their QA program. Many of their models are built using the GoldSim modeling platform, for which their models are well-known for their level of transparency, documentation, and QA traceability. This level of QA is also applied to their process-level models. Neptune has taken some lessons learned from the extremely rigorous QA program required for the Yucca Mountain Project (YMP), and imposed the YMP requirements of complete traceability, transparency, and reproducibility, but avoided the inflexibility of a QA program that likely contributed to the suspect e-mails in 2005 that resulted in loss of trust in the YMP, and in nuclear waste disposal (and nuclear power) in general. There are far too many examples of loss of public trust due to poor QA that could have been easily avoided with a simple and straight-forward QA program combined with stakeholder engagement. The purpose of this paper is to share features of Neptune's Radiological Performance Assessment program QA program, and some of their QA related lessons learned over the past 27 years of QA for complex environmental modeling. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗
DOE OSTI2022

Cloud-Control of Legacy Building Automation System: A case study

As Internet of Things devices and cloud-based platforms become more mature, Energy Management and Information Systems (EMIS) are increasingly gaining momentum in the building industry. In large commercial buildings, Fault-Detection and Diagnostic (FDD) and energy information systems (EIS) are now established technologies with tens of providers and thousands of deployment sites across North America. The new frontier for the EMIS technology is now represented by control systems that use advanced system optimization (ASO) methods to improve the operations of the HVAC system. Given the complexity of the integration of such systems with the existing building automation systems (BAS) and the higher risk involved with direct control of the HVAC, these systems are still emerging in the market. This paper presents the results of a project in which a start-up company partnered with a research institution to develop a cloud-based software EMIS solution and deployed it in a university campus in California. The software system included advanced sensing, data acquisition, storage and advanced control and analytics applications developed on top of the native BAS. The new platform controls ten buildings on the campus and the FDD and the ASO applications deployed on this platform were able to generate energy savings of up to 35% and 25% in certain buildings for each functionality respectively. Where the platform did not save energy, it improved building service (air quality). Lessons learned include the importance of collaborating with and training the building operators and evaluating whether the legacy system can work reliably with the new technology.

Prakash, Anand Krishnan↗
NASA NTRS2024

Safety Assessment of Conformance Monitoring for Situational Awareness in UTM Operations

This report presents a systematic approach to evaluating the safety benefit of utilizing strategic deconfliction and Conformance Monitoring for Situational Awareness to manage the traffic that is comprised of both conforming aircraft and contingent aircraft. First, we developed a Monte-Carlo-based simulation platform that generates a range of nominal flight paths, a set of flight paths associated with contingent aircraft, encounters between conforming and contingent aircraft, and simulates aircraft's reported positions within conformance bounds. Next, we derived the mathematical equations for quantifying the level of safety, represented by the probability of mid-air collision per flight hour, using the flight data from the simulation platform. Finally, we compared the safety benefit of CMSA services in scenarios generated at different traffic densities and rates of contingent aircraft. This study suggests that strategic deconfliction and CMSA services can collaboratively handle traffic mixed with conforming and contingent aircraft more safely than strategic deconfliction alone.

CMSA↗
DOE OSTI2018

Striped Data Server for Scalable Parallel Data Analysis

A columnar data representation is known to be an efficient way for data storage, specifically in cases when the analysis is often done based only on a small fragment of the available data structures. A data representation like Apache Parquet is a step forward from a columnar representation, which splits data horizontally to allow for easy parallelization of data analysis. Based on the general idea of columnar data storage, working on the [LDRD Project], we have developed a striped data representation, which, we believe, is better suited to the needs of High Energy Physics data analysis. A traditional columnar approach allows for efficient data analysis of complex structures. While keeping all the benefits of columnar data representations, the striped mechanism goes further by enabling easy parallelization of computations without requiring special hardware. We will present an implementation and some performance characteristics of such a data representation mechanism using a distributed no-SQL database or a local file system, unified under the same API and data representation model. The representation is efficient and at the same time simple so that it allows for a common data model and APIs for wide range of underlying storage mechanisms such as distributed no-SQL databases and local file systems. Striped storage adopts Numpy arrays as its basic data representation format, which makes it easy and efficient to use in Python applications. The Striped Data Server is a web service, which allows to hide the server implementation details from the end user, easily exposes data to WAN users, and allows to utilize well known and developed data caching solutions to further increase data access efficiency. We are considering the Striped Data Server as the core of an enterprise scale data analysis platform for High Energy Physics and similar areas of data processing. As a result, we have been testing this architecture with a 2TB dataset from a CMS dark matter search and plan to expand it to multiple 100 TB or even PB scale. We will present the striped format, Striped Data Server architecture and performance test results.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗