Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data analysis software”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

BatAnalysis - A Comprehensive Python Pipeline for Swift BAT Survey Analysis

The Swift Burst Alert Telescope (BAT) is a coded-aperture gamma-ray instrument with a large field of view that primarily operates in survey mode when it is not triggering on transient events. The survey data consist of 80-channel detector plane histograms that accumulate photon counts over periods of at least 5 minutes. These histograms are processed on the ground and are used to produce the survey data set between 14 and 195 keV. Survey data comprise >90% of all BAT data by volume and allow for the tracking of long-term light curves and spectral properties of cataloged and uncataloged hard X-ray sources. Until now, the survey data set has not been used to its full potential due to the complexity associated with its analysis and the lack of easily usable pipelines. Here, we introduce the BatAnalysis Python package, a wrapper for HEASoftpy, which provides a modern, open-source pipeline to process and analyze BAT survey data. BatAnalysis allows members of the community to use BAT survey data in more advanced analyses of astrophysical sources, including pulsars, pulsar wind nebula, active galactic nuclei, and other known/unknown transient events that may be detected in the hard X-ray band. We outline the steps taken by the Python code and exemplify its usefulness and accuracy by analyzing survey data of the Crab Nebula, NGC 2992, and a previously uncataloged MAXI transient. The BatAnalysis package allows for ~18 yr of BAT survey data to be used in a systematic way to study a large variety of astrophysical sources.

79 ASTRONOMY AND ASTROPHYSICS↗

A Short Survey on the State of the Art in Architectures and Platforms for Large Scale Data Analysis and Knowledge Discovery from Data

Intended as a survey for practicing architects and researchers seeking an overview of the state-of-the-art architectures for data analysis, this paper provides an overview of the emerg- ing data management and analytic platforms including par- allel databases, Hadoop-based systems, High Performance Computing (HPC) platforms and platforms popularly re- ferred to as NoSQL platforms. Platforms are presented based on their relevance, analysis they support and the data organization model they support.

99 GENERAL AND MISCELLANEOUS↗

Arrowland v1.0

"Arrowland" is a web-based software application primarily for mapping, integrating and visualizing a variety of metabolism data of living organisms, including but not limited to metabolomics, proteomics, transcriptomics and fluxomics. This software application makes multi-omics data analysis intuitive and interactive. It improves data sharing and communication by enabling users to visualize their omics data using a web browser (on a PC or mobile device). It increases user's productivity by simplifying multi-omics data analysis using well developed maps as a guide. Users using this tool can gain insights into their data sets that would be difficult or even impossible to tease out by looking at raw number, or using their currently existing toolchains to generate static single-use maps. Arrowland helps users save time by visualizing relative changes in different conditions or over time, and helps users to produce more significant insights faster. Preexisting maps decrease the learning curve for beginners in the omics field. Sets of multi-omics data are presented in the browser, as a two-dimensional flowchart resembling a map, with varying levels of detail information, based on the scaling of the map. Users can pan and zoom to explore different maps, compare maps, upload their own research data sets onto desired maps, alter map appearance in ways that facilitate interpretation, visualization and analysis of the given data, and export data, reports and actionable items to help the user initiative.

BIRKEL, GARRETT↗

HEP Analysis Ecosystem Workshop Report

Over the past 20 years the HEP community has developed and gravitated around an analysis ecosystem centered on ROOT. ROOT and its ecosystem both dominate HEP analysis and impact the full event processing chain, providing foundation libraries, I/O services etc. that have prevalence in the field. The analysis tools landscape is however evolving in ways that can have a durable impact on the analysis ecosystem and a strong influence on the analysis and core software landscape a decade from now, a timescale currently in an intensive planning round with the HEP Software Foundation(HSF) Community White Paper process. Data intensive analysis is growing in importance in other sciences and in the wider world. Powerful tools and new development initiatives, both within our field and in the wider open source community, have emerged. Creative developers have an ever more powerful open source toolkit available and are applying it towards innovations that leverage both open source and the ROOT ecosystem. ROOT itself is approaching a major re-engineering with ROOT 7, leveraging the powerful evolution of C++, with a major overhaul in its interfaces as seen both by its users and by satellite tools in the ROOT ecosystem.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Framework for Analysis and Visualization of Simulation Data (VizAly)

As supercomputing resources increase, cosmological scientists are able to run more detailed and larger simulations generating massive amounts of data. Analyzing these simulations with an available open-source toolkit is important for collaborative Department of Energy scientific discovery across labs, universities, and other partners. Developed software as a part of this collection include: comparing data with other existing simulations, verifying and validating results with observation databases, new halo finder algorithms, and using analytical tools to get insights into the physics of the cosmological universe. The goal of this software project is to provide a set of open-source libraries, tools, and packages for large-scale cosmology that allows scientists to visualize, analyze, and compare large-scale simulation and observational data sets. Developed software will provide a variety of methods for processing, visualization, and analysis of astronomical observation and cosmological simulation data. These tools are intended for deployment on multiple scientific computing platforms, including but not limited to personal computers, cloud computing, experimental sites (telescopes) and high performance supercomputers.

Wendelberger, Joanne↗

Earth and Space Science Informatics Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science

Abstract This article is composed of three independent commentaries about the state of Integrated, Coordinated, Open, Networked (ICON) principles (Goldman, et al., 2021b, https://doi.org/10.1029/2021EO153180 ) in Earth and Space Science Informatics (ESSI) and includes discussion on the opportunities and challenges of adopting them. Each commentary focuses on a different topic: (Section 2) Global collaboration, cyberinfrastructure, and data sharing; (Section 3) Machine learning for multiscale modeling; (Section 4) Aerial and satellite remote sensing for advancing Earth system model development by integrating field and ancillary data. ESSI addresses data management practices, computation and analysis, and hardware and software infrastructure. Our role in ICON science therefore involves collaborative work to assess, design, implement, and promote practices and tools that enable effective data management, discovery, integration, and reuse for interdisciplinary work in Earth and space science disciplines. Networks of diverse people with expertise across Earth, space, and data science disciplines are essential for efficient and ethical exchanges of findable, accessible, interoperable, and reusable (FAIR) research products and practices. Our challenge is then to coordinate the development of standards, curation practices, and tools that enable integrating and reusing multiple data types, software, multi‐scale models, and machine learning approaches across disciplines in a way that is as open and/or FAIR as ethically possible. This is a major endeavor that could greatly increase the pace and potential of interdisciplinary scientific discovery.

58 GEOSCIENCES↗

Stepwise v. 2.21

The statistical code, STEPWISE, evaluates variable importance by developing regression models between the observed response and input variables using either a forward, backward, or stepwise regression procedure on the raw or ranked data. The software will be used for performing sensitivity analysis on probabilistic results. Stepwise creates linear (or monotonic) regression of the output of interest using a set of input data in a stepwise fashion (i.e., adding one variable at a time). The program estimates parameters such as regression and standardized regressions coefficients for each selected input variables as well as the coefficient of determination of the resulting model

Sallaberry, CedricJean-Marie↗

Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1

Handling large streaming data is essential for various applications such as network traffic analysis, social networks, energy cost trends, and environment modeling. However, it is in general intractable to store, compute, search, and retrieve large streaming data. This software addresses a fundamental issue, which is to reduce the size of large streaming data and still obtain accurate statistical analysis. As an example, when a high-speed network such as 100 Gbps network is monitored, the collected measurement data rapidly grows so that polynomial time algorithms (e.g., Gaussian processes) become intractable. One possible solution to reduce the storage of vast amounts of measured data is to store a random sample, such as one out of 1000 network packets. However, such static sampling methods (linear sampling) have drawbacks: (1) it is not scalable for high-rate streaming data, and (2) there is no guarantee of reflecting the underlying distribution. In this software, we implemented a dynamic sampling algorithm, based on the recent technology from the relational dynamic bayesian online locally exchangeable measures, that reduces the storage of data records in a large scale, and still provides accurate analysis of large streaming data. The software can be used for both online and offline data records.

Sim, Alex↗

Automation of particulate characterization

Foreign particles residing on the field carrying surface of accelerator cavities are a known mechanism for field emission. Developing the methods and tools for collecting and characterizing particles found in an accelerator enables process development towards field emission free SRF cavities. Methods are presented for sampling assemblies, components, processes, and environmental conditions utilizing forensic techniques with specialized tooling. Sampling activities to date have produced an inventory of over 850 GSR spindles. Traditional SEM + EDS analysis of this volume of spindles is challenged by labor investment, spindle sampling methods, and the subsequent data pipeline which ultimately results in a statically inadequate dataset for any particulate distribution characterization. A complete systematic analysis of the spindles is enabled by third party software controlling SEM automation for EDS data acquisition. Details of spindle creation, collection equipment, component sampling, automating particle assessment, and data analysis used to characterize samples from beamline elements in CEBAF are presented.

Spradlin, Josh↗

OpenMSI: A High-Performance Web-Based Platform for Mass Spectrometry Imaging

Mass spectrometry imaging (MSI) enables researchers to directly probe endogenous molecules directly within the architecture of the biological matrix. Unfortunately, efficient access, management, and analysis of the data generated by MSI approaches remain major challenges to this rapidly developing field. Despite the availability of numerous dedicated file formats and software packages, it is a widely held viewpoint that the biggest challenge is simply opening, sharing, and analyzing a file without loss of information. Here we present OpenMSI, a software framework and platform that addresses these challenges via an advanced, high-performance, extensible file format and Web API for remote data access (http://openmsi.nersc.gov). The OpenMSI file format supports storage of raw MSI data, metadata, and derived analyses in a single, self-describing format based on HDF5 and is supported by a large range of analysis software (e.g., Matlab and R) and programming languages (e.g., C++, Fortran, and Python). Careful optimization of the storage layout of MSI data sets using chunking, compression, and data replication accelerates common, selective data access operations while minimizing data storage requirements and are critical enablers of rapid data I/O. The OpenMSI file format has shown to provide >2000-fold improvement for image access operations, enabling spectrum and image retrieval in less than 0.3 s across the Internet even for 50 GB MSI data sets. To make remote high-performance compute resources accessible for analysis and to facilitate data sharing and collaboration, we describe an easy-to-use yet powerful Web API, enabling fast and convenient access to MSI data, metadata, and derived analysis results stored remotely to facilitate high-performance data analysis and enable implementation of Web based data sharing, visualization, and analysis.

59 BASIC BIOLOGICAL SCIENCES↗

Implementation Aspects of Smart Grids Cyber-Security Cross-Layered Framework for Critical Infrastructure Operation

Communication networks in power systems are a major part of the smart grid paradigm. It enables and facilitates the automation of power grid operation as well as self-healing in contingencies. Such dependencies on communication networks, though, create a roam for cyber-threats. An adversary can launch an attack on the communication network, which in turn reflects on power grid operation. Attacks could be in the form of false data injection into system measurements, flooding the communication channels with unnecessary data, or intercepting messages. Using machine learning-based processing on data gathered from communication networks and the power grid is a promising solution for detecting cyber threats. In this paper, a co-simulation of cyber-security for cross-layer strategy is presented. The advantage of such a framework is the augmentation of valuable data that enhances the detection as well as identification of anomalies in the operation of the power grid. The framework is implemented on the IEEE 118-bus system. The system is constructed in Mininet to simulate a communication network and obtain data for analysis. A distributed three controller software-defined networking (SDN) framework is proposed that utilizes the Open Network Operating System (ONOS) cluster. According to the findings of our suggested architecture, it outperforms a single SDN controller framework by a factor of more than ten times the throughput. This provides for a higher flow of data throughout the network while decreasing congestion caused by a single controller’s processing restrictions. Furthermore, our CECD-AS approach outperforms state-of-the-art physics and machine learning-based techniques in terms of attack classification. The performance of the framework is investigated under various types of communication attacks.

cross-layered↗

Challenges and solutions for the analysis of in situ, in crystallo micro-spectrophotometric data

The particular challenge of the analysis of optical absorption and Raman spectroscopic data measured from protein crystals and how the SLS-APE software toolbox supports scientists in dealing with such data is described. Combining macromolecular crystallography with in crystallo micro-spectrophotometry yields valuable complementary information on the sample, including the redox states of metal cofactors, the identification of bound ligands and the onset and strength of undesired photochemistry, also known as radiation damage. However, the analysis and processing of the resulting data differs significantly from the approaches used for solution spectrophotometric data. The varying size and shape of the sample, together with the suboptimal sample environment, the lack of proper reference signals and the general influence of the X-ray beam on the sample have to be considered and carefully corrected for. In the present article, how to characterize and treat these sample-dependent artefacts in a reproducible manner is discussed and the SLS-APEin situ, in crystallo optical spectroscopy data-analysis toolbox is demonstrated.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Basic Energy Sciences Exascale Requirements Review. An Office of Science review sponsored jointly by Advanced Scientific Computing Research and Basic Energy Sciences, November 3-5, 2015, Rockville, Maryland

Computers have revolutionized every aspect of our lives. Yet in science, the most tantalizing applications of computing lie just beyond our reach. The current quest to build an exascale computer with one thousand times the capability of today’s fastest machines (and more than a million times that of a laptop) will take researchers over the next horizon. The field of materials, chemical reactions, and compounds is inherently complex. Imagine millions of new materials with new functionalities waiting to be discovered — while researchers also seek to extend those materials that are known to a dizzying number of new forms. We could translate massive amounts of data from high precision experiments into new understanding through data mining and analysis. We could have at our disposal the ability to predict the properties of these materials, to follow their transformations during reactions on an atom-by-atom basis, and to discover completely new chemical pathways or physical states of matter. Extending these predictions from the nanoscale to the mesoscale, from the ultrafast world of reactions to long-time simulations to predict the lifetime performance of materials, and to the discovery of new materials and processes will have a profound impact on energy technology. In addition, discovery of new materials is vital to move computing beyond Moore’s law. To realize this vision, more than hardware is needed. New algorithms to take advantage of the increase in computing power, new programming paradigms, and new ways of mining massive data sets are needed as well. This report summarizes the opportunities and the requisite computing ecosystem needed to realize the potential before us. In addition to pursuing new and more complete physical models and theoretical frameworks, this review found that the following broadly grouped areas relevant to the U.S. Department of Energy (DOE) Office of Advanced Scientific Computing Research (ASCR) would directly affect the Basic Energy Sciences (BES) mission need. Simulation, visualization, and data analysis are crucial for advances in energy science and technology. Revolutionary mathematical, software, and algorithm developments are required in all areas of BES science to take advantage of exascale computing architectures and to meet data analysis, management, and workflow needs. In partnership with ASCR, BES has an emerging and pressing need to develop new and disruptive capabilities in data science. More capable and larger high-performance computing (HPC) and data ecosystems are required to support priority research in BES. Continued success in BES research requires developing the next-generation workforce through education and training and by providing sustained career opportunities.

97 MATHEMATICS AND COMPUTING↗

Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis

X‐ray scattering experiments using free electron lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID‐19 viral proteins). XFEL experiments are a challenge to computing in two ways: (i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; (ii) data‐collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min—sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS‐NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Completing data analysis within 10 min is a first for XFEL experiments and an important milestone if we are to keep up with data‐collection trends.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis

X-ray scattering experiments using Free Electron Lasers (XFELs) are a powerful tool to determine the molecular structure and function of unknown samples (such as COVID-19 viral proteins). XFEL experiments are a challenge to computing in two ways: i) due to the high cost of running XFELs, a fast turnaround time from data acquisition to data analysis is essential to make informed decisions on experimental protocols; ii) data collection rates are growing exponentially, requiring new scalable algorithms. Here we report our experiences analyzing data from two experiments at the Linac Coherent Light Source (LCLS) during September 2020. Raw data were analyzed on NERSC's Cori XC40 system, using the Superfacility paradigm: our workflow automatically moves raw data between LCLS and NERSC, where it is analyzed using the software package CCTBX. We achieved real time data analysis with a turnaround time from data acquisition to full molecular reconstruction in as little as 10 min -- sufficient time for the experiment's operators to make informed decisions. By hosting the data analysis on Cori, and by automating LCLS-NERSC interoperability, we achieved a data analysis rate which matches the data acquisition rate. Furthermore, completing data analysis with 10 mins is a first for XFEL experiments and an important milestone if we are to keep up with data collection trends.

Blaschke, Johannes P.↗

In-Field Performance Testing of the Fork Detector for Quantitative Spent Fuel Verification

Expanding spent fuel dry storage activities worldwide are increasing demands on safeguards authorities that perform inspections. The European Atomic Energy Community (EURATOM) and the International Atomic Energy Agency (IAEA) require measurements to verify declarations when spent fuel is transferred to difficult-to-access locations, such as dry storage casks and the repositories planned in Finland and Sweden. EURATOM makes routine use of the Fork detector to obtain gross gamma and total neutron measurements during spent fuel inspections. Data analysis is performed by modules in the integrated Review and Analysis Program (iRAP) software, developed jointly by EURATOM and the IAEA. Under the framework of the US Department of Energy–EURATOM cooperation agreement, a module for automated Fork detector data analysis has been developed by Oak Ridge National Laboratory (ORNL) using the ORIGEN code from the SCALE code system and implemented in iRAP. EURATOM and ORNL recently performed measurements on 30 spent fuel assemblies at the Swedish Central Interim Storage Facility for Spent Nuclear Fuel (Clab), operated by the Swedish Nuclear Fuel and Waste Management Company (SKB). The measured assemblies represent a broad range of fuel characteristics. Neutron count rates for 15 measured pressurized water reactor assemblies are predicted with an average relative standard deviation of 4.6%, and gamma signals are predicted on average within 2.6% of the measurement. The 15 measured boiling water reactor assemblies exhibit slightly larger deviations of 5.2% for the gamma signals and 5.7% for the neutron count rates, compared to measurements. These findings suggest that with improved analysis of the measurement data, existing instruments can provide increased verification of operator declarations of the spent fuel and thereby also provide greater ability to confirm integrity of an assembly. These results support the application of the Fork detector as a fully quantitative spent fuel verification technique.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Mercat

Mercat is a highly scalable property software package for robust of analysis of features in next-generation sequencing data and observed unique peptides from metaproteomic data. Mercat is offered in python 3.5/anaconda3, is multiparellel and is easily installed using bioconda/conda recipes. Mercat inputs include assembled contigs, raw sequence reads from any platform, and unique peptide files obtained from proteomics with feature abundance counts tables. Mercat is the only software available that allows for direct analysis of data properties without a data-dependent search tools such as BLAST or diamond for compositional analysis of whole community shotgun sequencing (e.g. metagenomes and/or metatranscriptomes) or metaproteomic data.

Source record↗

Mercat

Mercat is a highly scalable property software package for robust of analysis of features in next-generation sequencing data and observed unique peptides from metaproteomic data. Mercat is offered in python 3.5/anaconda3, is multiparellel and is easily installed using bioconda/conda recipes. Mercat inputs include assembled contigs, raw sequence reads from any platform, and unique peptide files obtained from proteomics with feature abundance counts tables. Mercat is the only software available that allows for direct analysis of data properties without a data-dependent search tools such as BLAST or diamond for compositional analysis of whole community shotgun sequencing (e.g. metagenomes and/or metatranscriptomes) or metaproteomic data.

Source record↗