Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46

Patience is a virtue: A data-driven analysis of rooftop solar PV permitting timelines in the United States

© 2020 The Authors Local permitting can ensure the safe installation and operation of rooftop solar photovoltaic (PV) systems. At the same time, burdensome local permitting processes and local variation in requirements may pose challenges to PV deployment. In this article, we explore new data on the durations between key steps in the PV permitting process in the United States. The data suggest that a typical customer can expect to wait around 25–100 days from permit application until an installed system passes inspection. Permit durations vary significantly across jurisdictions, due in part to differences in local permitting policies. However, permit durations vary as significantly within jurisdictions as across them, in part due to significant variation across installers, suggesting that installer strategies and practices play an important role in permitting timelines. Permit durations have declined over time, reflecting progress from permit streamlining policies and jurisdiction learning-by-doing, though durations have stabilized in recent years. The data suggest that typical PV customers still face long and uncertain permitting timelines in the United States.

O'Shaughnessy, E↗

Visualization and data analysis-current state and exascale challenges

Talk about Visualization and Data Analysis Current State and Exascale challenges. The goal is to update with colleagues our current status in our research. What challenges we need to face, and what future possibilities. Our goal is to propose to approach the problems with the visualization approach operating on the supercomputing platform. This presentation is about the L2 Milestone, we intend to discuss further possibilities of enhancing our results and optimizing our solutions.

97 MATHEMATICS AND COMPUTING↗

Echo™ User Manual

Echo™ is a MATLAB-based software package designed for robust and scalable analysis of complex data workflows. An alternative to tedious, error-prone conventional processes, Echo is based on three transformative principles for data analysis: self-describing data, name-based indexing, and dynamic resource allocation. The software takes an object-oriented approach to data analysis, intimately connecting measurement data with associated metadata. Echo operations in an analysis workflow automatically track and merge metadata and computation parameters to provide a complete history of the process used to generate final results, while automated figure and report generation tools eliminate the potential to mislabel those results. History reporting and visualization methods provide straightforward auditability of analysis processes. Furthermore, name-based indexing on metadata greatly improves code readability for analyst collaboration and reduces opportunities for errors to occur. Echo efficiently manages large data sets using a framework that seamlessly allocates resources such that only the necessary computations to produce a given result are executed. Echo provides a versatile and extensible framework, allowing advanced users to add their own tools and data classes tailored to their own specific needs. Applying these transformative principles and powerful features, Echo greatly improves analyst efficiency and quality of results in many application areas.

97 MATHEMATICS AND COMPUTING↗

CMS Analysis and Data Reduction with Apache Spark

Experimental Particle Physics has been at the forefront of analyzing the world's largest datasets for decades. The HEP community was among the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems for distributed data processing, collectively called "Big Data" technologies have emerged from industry and open source projects to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and tools, promising a fresh look at analysis of very large datasets that could potentially reduce the time-to-physics with increased interactivity. Moreover these new tools are typically actively developed by large communities, often profiting of industry resources, and under open source licensing. These factors result in a boost for adoption and maturity of the tools and for the communities supporting them, at the same time helping in reducing the cost of ownership for the end-users. In this talk, we are presenting studies of using Apache Spark for end user data analysis. We are studying the HEP analysis workflow separated into two thrusts: the reduction of centrally produced experiment datasets and the end-analysis up to the publication plot. Studying the first thrust, CMS is working together with CERN openlab and Intel on the CMS Big Data Reduction Facility. The goal is to reduce 1 PB of official CMS data to 1 TB of ntuple output for analysis. We are presenting the progress of this 2-year project with first results of scaling up Spark-based HEP analysis. Studying the second thrust, we are presenting studies on using Apache Spark for a CMS Dark Matter physics search, comparing Spark's feasibility, usability and performance to the ROOT-based analysis.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Scenarios for long-term analysis

Data Preservation and Long-Term Analysis of High Energy Physics (HEP) Experiments data is described and summarized in this talk. The summary covers information presented at the First Workshop on Data Preservation and Long-Term Analysis. Experiments representing e{sup +}e{sup -} collisions (LEP, B Factories and CLEO), ep collisions (H1 and ZEUS), p{bar p} collisions (CDF and D0) and others presented interesting information related to utilizing the large datasets collected over many years at these HEP facilities. Many questions and issues remain to be explored.

99 GENERAL AND MISCELLANEOUS↗

Integrated Data Collection Analysis (IDCA) Program - RDX Type II Class 5 Standard, Data Set 1

This document describes the results of the first reference sample material—RDX Type II Class 5—examined in the proficiency study for small-scale safety and thermal (SSST) testing of explosive materials for the Integrated Data Collection Analysis (IDCA) Program. The IDCA program is conducting proficiency testing on homemade explosives (HMEs). The reference sample materials are being studied to establish the accuracy of traditional explosives safety testing for each performing laboratory. These results will be used for comparison to results from testing HMEs. This effort, funded by the Department of Homeland Security (DHS), ultimately will put the issues of safe handling of these materials in perspective with standard military explosives. The results of the study will add SSST testing results for a broad suite of different HMEs to the literature, potentially suggest new guidelines and methods for HME testing, and possibly establish what are the needed accuracies in SSST testing to develop safe handling practices. Described here are the results for impact, friction, electrostatic discharge, and scanning calorimetry analysis of a reference sample of RDX Type II Class 5. The results from each participating testing laboratory are compared using identical test material and preparation methods wherever possible. Note, however, the test procedures differ among the laboratories. These results are then compared to historical data from various sources. The performers involved are Lawrence Livermore National Laboratory (LLNL), Los Alamos National Laboratory (LANL), Air Force Research Laboratory/ RXQL (AFRL), Indian Head Division, Naval Surface Warfare Center, (IHD-NSWC), and Sandia National Laboratories (SNL). These tests are conducted as a proficiency study in order to establish some consistency in test protocols, procedures, and experiments and to understand how to compare results when test protocols are not identical.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Strym: A Python Package for Real-time CAN Data Logging, Analysis and Visualization to Work with USB-CAN Interface

In this report, we describe a data analysis tool developed for decoding and analyzing vehicle data obtained from a passenger vehicle’s onboard controller area network (CAN) bus. The tool developed in this paper provides a timeseries framework to perform domain-specific analysis at scale when interpreting data from a vehicle or a collection of vehicles in light of how to design intelligent vehicle applications. The tool, called Strym, exploits the CAN bus mechanism of modern vehicles to capture data using commercially available CAN-to-USB hardware Comma.ai Panda devices, managed through open-source software Libpanda. Strym permits the decoding of vendor-specific CAN messages in a vehicle-agnostic manner. Through this, a researcher can characterize data throughput, assess data quality, and perform analyses. Such analyses are useful in a number of research such as studying human driving behavior in mixed-autonomy, new driver models, rare-event detection, traffic flow estimation, and custom control of vehicles.

Performance evaluation, Smart cities, Intelligent ↗

A browser-based tool for visualization and analysis of diffusion MRI data

Human neuroscience research faces several challenges with regards to reproducibility. While scientists are generally aware that data sharing is important, it is not always clear how to share data in a manner that allows other labs to understand and reproduce published findings. Here we report a new open source tool, AFQ-Browser, that builds an interactive website as a companion to a diffusion MRI study. Because AFQ-Browser is portable—it runs in any web-browser—it can facilitate transparency and data sharing. Moreover, by leveraging new web-visualization technologies to create linked views between different dimensions of the dataset (anatomy, diffusion metrics, subject metadata), AFQ-Browser facilitates exploratory data analysis, fueling new discoveries based on previously published datasets. In an era where Big Data is playing an increasingly prominent role in scientific discovery, so will browser-based tools for exploring high-dimensional datasets, communicating scientific discoveries, aggregating data across labs, and publishing data alongside manuscripts.

60 APPLIED LIFE SCIENCES↗

Continuous Monitoring and On-line Analysis of Operational Dose Rates: Tools to Further Mitigate Radiation Risks

Along with passive and active engineering and administrative controls usually implemented in radiation protection programs at different facilities, continuous monitoring and on-line analysis of data measured by the radiation detectors at the workplace and in the environment can be considered as an additional tool used to further mitigate radiation risks. Many monitoring systems on the market today allow connecting radiation area monitors into a network, and reading and accumulating data continuously in a database. The point of this presentation is to bring attention to the fact that such accumulated information can be analyzed and used in many respects to improve reliability and functionality of the monitoring and control systems. A simple time history of background readings from a radiation detector can be used to evaluate the stability of the detector performance. Data recorded during normal facility operations may serve to establish a pattern of acceptable dose rates in such detector and allow to better detect off-normal and unstable modes of the facility operation before they reach hardware trip levels. Facility operators and users may utilize such monitoring systems to optimize operations and minimize their radiological impact. Implementation and examples of use of the RADMON radiation monitoring and data analysis system at Jefferson Lab is presented.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Performance Analysis and Optimization for Scientific Data Workloads

Scientific data generated at experimental and observational facilities are increasingly being processed on large-scale compute systems. Most of the experimental data analysis workflows are not designed or implemented to run on large scale environments and take full advantage of HPC compute and storage resources. These applications are unlike the traditional tightly-coupled scientific applications and hence face significant performance and scalability challenges as the volume of data increases exponentially. In this paper, we conduct a performance and scalability analysis for experimental analysis applications and workflows operating on data from light sources. Our analysis detects and quantifies I/O performance, scalability and runtime bottlenecks for three data analysis applications that run on NERSC resources. Based on our analysis we propose and implement a set of optimizations that lead to reducing the amount of time spent on I/O operations by almost 90%.

97 MATHEMATICS AND COMPUTING↗

Echo: Data and Analysis Management [Slides]

Echo software capabilities are described. Echo provides a comprehensive suite of tools for data wrangling, management, and analysis. It is an object-oriented approach to data analysis designed for robust and scalable execution of complex analysis workflows written in MATLAB.

97 MATHEMATICS AND COMPUTING↗

A data-driven analysis of the heavy quark transport coefficient

Using a Bayesian model-to-data analysis, we estimate the temperature dependence of the heavy quark diffusion coefficients by calibrating to the experimental data of D-meson R AA and v 2 in AuAu collisions ($\sqrt{s}$NN = 200 GeV) and PbPb collisions ($\sqrt{s}$NN = 2.76 TeV) [1]. The spatial diffusion coefficient D s 2πT is found to be mostly constraint around (1.3 1.5)T c and is compatible with lattice QCD calculations. Here we demonstrate the capability of our improved Langevin model to simultaneously describe the R AA and v 2 at both RHIC and the LHC energies, as well as the feasibility to apply a Bayesian analysis to quantitatively study the heavy flavor transport in heavy-ion collisions.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

A Greenhouse-Gas Information System: Monitoring and Validating Emissions Reporting and Mitigation

This study and report focus on attributes of a greenhouse-gas information system (GHGIS) needed to support MRV&V needs. These needs set the function of such a system apart from scientific/research monitoring of GHGs and carbon-cycle systems, and include (not exclusively): the need for a GHGIS that is operational, as required for decision-support; the need for a system that meets specifications derived from imposed requirements; the need for rigorous calibration, verification, and validation (CV&V) standards, processes, and records for all measurement and modeling/data-inversion data; the need to develop and adopt an uncertainty-quantification (UQ) regimen for all measurement and modeling data; and the requirement that GHGIS products can be subjected to third-party questioning and scientific scrutiny. This report examines and assesses presently available capabilities that could contribute to a future GHGIS. These capabilities include sensors and measurement technologies; data analysis and data uncertainty quantification (UQ) practices and methods; and model-based data-inversion practices, methods, and their associated UQ. The report further examines the need for traceable calibration, verification, and validation processes and attached metadata; differences between present science-/research-oriented needs and those that would be required for an operational GHGIS; the development, operation, and maintenance of a GHGIS missions-operations center (GMOC); and the complex systems engineering and integration that would be required to develop, operate, and evolve a future GHGIS.

54 ENVIRONMENTAL SCIENCES↗

A Greenhouse-Gas Information System: Monitoring and Validating Emissions Reporting and Mitigation

Current GHG-mitigating regimes, whether internationally agreed or self-imposed, rely on the aggregation of self-reported data, with limited checks for consistency and accuracy, for monitoring. As nations commit to more stringent GHG emissions-mitigation actions and as economic rewards or penalties are attached to emission levels, self-reported data will require independent confirmation that they are accurate and reliable, if they are to provide the basis for critical choices and actions that may be required. Supporting emissions-mitigation efforts and agreements, as well as monitoring energy- and fossil-fuel intensive national and global activities would be best achieved by a process of: (1) monitoring of emissions and emission-mitigation actions, based, in part, on, (2) (self-) reporting of pertinent bottom-up inventory data, (3) verification that reported data derive from and are consistent with agreed-upon processes and procedures, and (4) validation that reported emissions and emissions-mitigation action data are correct, based on independent measurements (top-down) derived from a suite of sensors in space, air, land, and, possibly, sea, used to deduce and attribute anthropogenic emissions. These data would be assessed and used to deduce and attribute measured GHG concentrations to anthropogenic emissions, attributed geographically and, to the extent possible, by economic sector. The validation element is needed to provide independent assurance that emissions are in accord with reported values, and should be considered as an important addition to the accepted MRV process, leading to a MRV&V process. This study and report focus on attributes of a greenhouse-gas information system (GHGIS) needed to support MRV&V needs. These needs set the function of such a system apart from scientific/research monitoring of GHGs and carbon-cycle systems, and include (not exclusively): the need for a GHGIS that is operational, as required for decision-support; the need for a system that meets specifications derived from imposed requirements; the need for rigorous calibration, verification, and validation (CV&V) standards, processes, and records for all measurement and modeling/data-inversion data; the need to develop and adopt an uncertainty-quantification (UQ) regimen for all measurement and modeling data; and the requirement that GHGIS products can be subjected to third-party questioning and scientific scrutiny. This report examines and assesses presently available capabilities that could contribute to a future GHGIS. These capabilities include sensors and measurement technologies; data analysis and data uncertainty quantification (UQ) practices and methods; and model-based data-inversion practices, methods, and their associated UQ. The report further examines the need for traceable calibration, verification, and validation processes and attached metadata; differences between present science-/research-oriented needs and those that would be required for an operational GHGIS; the development, operation, and maintenance of a GHGIS missions-operations center (GMOC); and the complex systems engineering and integration that would be required to develop, operate, and evolve a future GHGIS. Present monitoring systems would be heavily relied on in any GHGIS implementation at the outset and would likely continue to provide valuable future contributions to GHGIS. However, present monitoring systems were developed to serve science/research purposes. This study concludes that no component or capability presently available is at the level of technological maturity and readiness required for implementation in an operational GHGIS today. However, purpose-designed and -built components could be developed and implemented in support of a future GHGIS. The study concludes that it is possible to develop and provide a capability-driven prototype GHGIS, as part of a Phase-1 effort, within three years from project-funding start, that would make use of and integrate existing sensing and system capabilities. As part of a Phase-2 effort, a requirements-driven, operational GHGIS could be developed, within ten years from project funding start. That schedule is driven by the development and long lead-times for some system components. The two efforts would be focused on different deliverables but could commence concurrently, to save time, if that was deemed desirable. We note that, developing and supporting an operational GHGIS will require a new approach and management, sustained funding and other support, as well as technical advances and development of purpose-built components that meet the requisite specifications. A functioning GHGIS will provide the basis for reasoned choices on how best to respond to rising GHG levels, especially when proposed U.S. actions are compared with or conditioned on the actions of other nations.

54 ENVIRONMENTAL SCIENCES↗

Why High Performance Visual Data Analytics is both Relevant and Difficult

Data visualization, as well as data analysis and data analytics, are all an integral part of the scientific process. Collectively, these technologies provide the means to gain insight into data of ever-increasing size and complexity. Over the past two decades, a substantial amount of visualization, analysis, and analytics R&D has focused on the challenges posed by increasing data size and complexity, as well as on the increasing complexity of a rapidly changing computational platform landscape. While some of this research focuses on solely on technologies, such as indexing and searching or novel analysis or visualization algorithms, other R&D projects focus on applying technological advances to specific application problems. Some of the most interesting and productive results occur when these two activities R&D and application are conducted in a collaborative fashion, where application needs drive R&D, and R&D results are immediately applicable to real world problems.

97 MATHEMATICS AND COMPUTING↗

Management, Analysis, and Visualization of Experimental and Observational Data – The Convergence of Data and Computing

Scientific user facilities—particle accelerators, telescopes, colliders, supercomputers, light sources, sequencing facilities, and more—operated by the U.S. Department of Energy (DOE) Office of Science (SC) generate ever increasing volumes of data at unprecedented rates from experiments, observations, and simulations. At the same time there is a growing community of experimentalists that require real-time data analysis feedback, to enable them to steer their complex experimental instruments to optimized scientific outcomes and new discoveries. Recent efforts in DOE-SC have focused on articulating the data-centric challenges and opportunities facing these science communities. Key challenges include difficulties coping with data size, rate, and complexity in the context of both real-time and post-experiment data analysis and interpretation. Solutions will require algorithmic and mathematical advances, as well as hardware and software infrastructures that adequately support data-intensive scientific workloads. This paper presents the summary findings of a workshop held by DOE-SC in September 2015, convened to identify the major challenges and the research that is needed to meet those challenges.

Bethel, E. Wes↗