Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,081 records · Page 60

Development of a fast framing detector for electron microscopy

A high frame rate detector system is described that enables fast real-time data analysis of scanning diffraction experiments in scanning transmission electron microscopy (STEM). This is an end-to-end development that encompasses the data producing detector, data transportation, and real-time processing of data. The detector will consist of a central pixel sensor that is surrounded by annular silicon diodes. Both components of the detector system will synchronously capture data at almost 100 kHz frame rate, which produces an approximately 400 Gb/s data stream. Low-level preprocessing will be implemented in firmware before the data is streamed from the National Center for Electron Microscopy (NCEM) to the National Energy Research Scientific Computing Center (NERSC). Live data processing, before it lands on disk, will happen on the Cori supercomputer and aims to present scientists with prompt experimental feedback. This online analysis will provide rough information of the sample that can be utilized for sample alignment, sample monitoring and verification that the experiment is set up correctly. Only a compressed version of the relevant data is then selected for more in-depth processing.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Exploring Classification of Topological Priors With Machine Learning for Feature Extraction

In many scientific endeavors, increasingly abstract representations of data allow for new interpretive methodologies and conceptualization of phenomena. For example, moving from raw imaged pixels to segmented and reconstructed objects allows researchers new insights and means to direct their studies toward relevant areas. Thus, the development of new and improved methods for segmentation remains an active area of research. With advances in machine learning and neural networks, scientists have been focused on employing deep neural networks such as U-Net to obtain pixel-level segmentations, namely, defining associations between pixels and corresponding/referent objects and gathering those objects afterward. Topological analysis, such as the use of the Morse-Smale complex to encode regions of uniform gradient flow behavior, offers an alternative approach: first, create geometric priors, and then apply machine learning to classify. This approach is empirically motivated since phenomena of interest often appear as subsets of topological priors in many applications. Using topological elements not only reduces the learning space but also introduces the ability to use learnable geometries and connectivity to aid the classification of the segmentation target. Here, in this article, we describe an approach to creating learnable topological elements, explore the application of ML techniques to classification tasks in a number of areas, and demonstrate this approach as a viable alternative to pixel-level classification, with similar accuracy, improved execution time, and requiring marginal training data.

97 MATHEMATICS AND COMPUTING↗

Nuclear Physics Science Network Requirements Workshop, May 2008 - Final Report

The Energy Sciences Network (ESnet) is the primary provider of network connectivity for the US Department of Energy Office of Science, the single largest supporter of basic research in the physical sciences in the United States of America. In support of the Office of Science programs, ESnet regularly updates and refreshes its understanding of the networking requirements of the instruments, facilities, scientists, and science programs that it serves. This focus has helped ESnet to be a highly successful enabler of scientific discovery for over 20 years. In May 2008, ESnet and the Nuclear Physics (NP) Program Office of the DOE Office of Science organized a workshop to characterize the networking requirements of the science programs funded by the NP Program Office. Most of the key DOE sites for NP related work will require significant increases in network bandwidth in the 5 year time frame. This includes roughly 40 Gbps for BNL, and 20 Gbps for NERSC. Total transatlantic requirements are on the order of 40 Gbps, and transpacific requirements are on the order of 30 Gbps. Other key sites are Vanderbilt University and MIT, which will need on the order of 20 Gbps bandwidth to support data transfers for the CMS Heavy Ion program. In addition to bandwidth requirements, the workshop emphasized several points in regard to science process and collaboration. One key point is the heavy reliance on Grid tools and infrastructure (both PKI and tools such as GridFTP) by the NP community. The reliance on Grid software is expected to increase in the future. Therefore, continued development and support of Grid software is very important to the NP science community. Another key finding is that scientific productivity is greatly enhanced by easy researcher-local access to instrument data. This is driving the creation of distributed repositories for instrument data at collaborating institutions, along with a corresponding increase in demand for network-based data transfers and the tools to manage those transfers effectively. Network reliability is also becoming more important as there is often a narrow window between data collection and data archiving when transfer and analysis can be done. The instruments do not stop producing data, so extended network outages can result in data loss due to analysis pipeline stalls. Finally, as the scope of collaboration continues to increase, collaboration tools such as audio and video conferencing are becoming ever more critical to the productivity of scientific collaborations.

97 MATHEMATICS AND COMPUTING↗

Asynchronous In Situ Connected-Components Analysis for Complex Fluid flows, In: ISAV '16 Proceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization

The simulation of multiscale physics is an important challenge for scientific computing. For this class of problem, large three-dimensional simulations are performed to advance scientific inquiry. On massively parallel computing systems, the volume of data generated by such approaches can become a productivity bottleneck if the raw data generated from the simulation is analyzed in a post-processing step. To address this, we present a physics-based framework for in situ data reduction that is theoretically grounded in multiscale averaging theory. We show how task parallelism can be exploited to concurrently perform a variety of analysis tasks with data-dependent costs, including the generation of iso-surfaces, morphological analyses, and connected components analysis. All analyses are performed in parallel using distributed memory and use the same domain decomposition as the simulation. A task management framework is constructed to leverage available parallelism within a node for analysis. The capabilities of the framework are to launch asynchronous analysis threads, manage dependencies between different tasks, promote data locality and minimize the impact of data transfers. The framework is applied to analyze GPU-based simulations of two-fluid-phase flow in porous media, generating a set of averaged measures that represents the overall system behavior. We demonstrate how the approach can be applied to perform physically-consistent analysis over fluid sub-regions determined from connected components analysis. Simulations performed on Oak Ridge National Lab's Titan supercomputer are profiled to demonstrate the performance of the associated multi-threaded in situ analysis approach for typical production simulation of two-fluid-phase flow.

Computer Science↗

Large-Scale Visualization of 3D Unstructured Groundwater Model Using Cave Automated Virtual Environment

The immersive three-dimensional (3D) virtual reality (VR) visualization of groundwater models allows us to deepen our understanding of aquifer systems and provide better solutions to present groundwater-related problems, such as groundwater recharge, water quality, and sustainability. Visualization assists in accurately developing groundwater models and revealing important subsurface features, including faulting, folding, and unconformity. However, assessing model accuracy poses challenges due to the complexity of geology and groundwater systems. This research demonstrates a workflow to visualize and analyze raw 3D unstructured groundwater model data using an immersive Cave Automated Virtual Environment (CAVE). To visualize the unstructured groundwater model data, the raw dataset is converted into interactive CAVE-compatible formats utilizing a set of tools: ParaView, Blender, and Unity. This enables researchers to immerse themselves in the data, identifying influential patterns and relationships. e resulting insights can inform the development of sophisticated machine-learning models for groundwater level prediction. The CAVE’s immersive capabilities allow intuitive exploration from various perspectives, providing a more holistic understanding of the factors affecting groundwater levels. These insights are crucial to improve predictive models. The CAVE results also facilitate collaborative analysis and have potential applications in training and education. is research demonstrates the value of immersive VR tools such as the CAVE for unraveling intricacies within high-dimensional scientific data to drive real-world forecasting and modeling applications.

54 ENVIRONMENTAL SCIENCES↗

dCache: The Storage System of Choice for Data-Intensive Applications

The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.

DCache↗

Data preservation at the Fermilab Tevatron

The Fermilab Tevatron collider's data-taking run ended in September 2011, yielding a dataset with rich scientific potential. The CDF and D0 experiments each have approximately 9 PB of collider and simulated data stored on tape. A large computing infrastructure consisting of tape storage, disk cache, and distributed grid computing for physics analysis with the Tevatron data is present at Fermilab. The Fermilab Run II data preservation project intends to keep this analysis capability sustained through the year 2020 and beyond. To achieve this goal, we have implemented a system that utilizes virtualization, automated validation, and migration to new standards in both software and data storage technology and leverages resources available from currently-running experiments at Fermilab. Lastly, these efforts have also provided useful lessons in ensuring long-term data access for numerous experiments, and enable high-quality scientific output for years to come.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Engineering and Scientific Applications: Using MatLab(Registered Trademark) for Data Processing and Visualization

MatLab(R) (MATrix LABoratory) is a numerical computation and simulation tool that is used by thousands Scientists and Engineers in many cou ntries. MatLab does purely numerical calculations, which can be used as a glorified calculator or interpreter programming language; its re al strength is in matrix manipulations. Computer algebra functionalities are achieved within the MatLab environment using "symbolic" toolbo x. This feature is similar to computer algebra programs, provided by Maple or Mathematica to calculate with mathematical equations using s ymbolic operations. MatLab in its interpreter programming language fo rm (command interface) is similar with well known programming languag es such as C/C++, support data structures and cell arrays to define c lasses in object oriented programming. As such, MatLab is equipped with most ofthe essential constructs of a higher programming language. M atLab is packaged with an editor and debugging functionality useful t o perform analysis of large MatLab programs and find errors. We belie ve there are many ways to approach real-world problems; prescribed methods to ensure foregoing solutions are incorporated in design and ana lysis of data processing and visualization can benefit engineers and scientist in gaining wider insight in actual implementation of their perspective experiments. This presentation will focus on data processing and visualizations aspects of engineering and scientific applicati ons. Specifically, it will discuss methods and techniques to perform intermediate-level data processing covering engineering and scientifi c problems. MatLab programming techniques including reading various data files formats to produce customized publication-quality graphics, importing engineering and/or scientific data, organizing data in tabu lar format, exporting data to be used by other software programs such as Microsoft Excel, data presentation and visualization will be discussed. The presentation will emphasize creating practIcal scripts (pro grams) that extend the basic features of MatLab TOPICS mclude (1) Ma trix and vector analysis and manipulations (2) Mathematical functions (3) Symbolic calculations & functions (4) Import/export data files (5) Program lOgic and flow control (6) Writing function and passing parameters (7) Test application programs

Sen, Syamal K.↗

Seismology in the Cloud: A New Streaming Workflow

Data-intensive research in seismology is experiencing a recent boom, driven in part by large volumes of available data and advances in the growing field of data science. However, there are significant barriers to processing large data volumes, such as long retrieval times from data repositories, complex data management, and limited computational resources. New tools and platforms have reduced the barriers to entry for scientific cluster computing, including the maturation of the commercial cloud as an accessible instrument for research. Here, we build a customized research cluster in the cloud to test a new workflow for large-scale seismic analysis, in which data are processed as a stream (retrieved on-the-fly and acted upon without storing), with data from the Incorporated Research Institutions for Seismology Data Management Center. We use this workflow to deploy a spectral peak detection algorithm over 5.6 TB of compressed continuous seismic data from 2074 stations of the USArray Transportable Array EarthScope network. Using a 50-node cluster in the cloud, we completed the noise survey in 80 hr, with an average data throughput of 1.7 GB per minute. By varying cluster sizes, we find the scaling of our analysis to be sublinear, due to a combination of algorithmic limitations and data center response times. The cloud-based streaming workflow represents an order-of-magnitude increase in acquisition and processing speed compared to a traditional download-store-process workflow, and offers the additional benefits of employing a flexible, accessible, and widely used computing architecture. It is limited, however, due to its reliance on Internet transfer speeds and data center service capacity, and may not work well for repeated analyses or those for which even higher data throughputs are needed. These research applications will require a new class of cloud-native approaches in which both data and analysis are in the cloud.

58 GEOSCIENCES↗

Calibrating spectral estimation for the LISA Technology Package with multichannel synthetic noise generation

The scientific objectives of the LISA Technology Package experiment on board of the LISA Pathfinder mission demand accurate calibration and validation of the data analysis tools in advance of the mission launch. The level of confidence required in the mission outcomes can be reached only by intensively testing the tools on synthetically generated data. A flexible procedure allowing the generation of a cross-correlated stationary noise time series was set up. A multichannel time series with the desired cross-correlation behavior can be generated once a model for a multichannel cross-spectral matrix is provided. The core of the procedure comprises a noise coloring, multichannel filter designed via a frequency-by-frequency eigendecomposition of the model cross-spectral matrix and a subsequent fit in the Z domain. The common problem of initial transients in a filtered time series is solved with a proper initialization of the filter recursion equations. The noise generator performance was tested in a two-dimensional case study of the closed-loop LISA Technology Package dynamics along the two principal degrees of freedom.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗

Using a Neural Network to Determine the Hatch Status of the AERI at the ARM North Slope of Alaska Site

The fore-optics of the Atmospheric Emitted Radiance Interferometer (AERI) are protected by an automated hatch to prevent precipitation from fouling the instrument's scene mirror (Knuteson et al. 2004). Limit switches connected with the hatch controller provide a signal of the hatch state: open, closed, undetermined (typically associated with the hatch being between fully open or fully closed during the instrument's sky view period), or an error condition. The instrument then records the state of the hatch with the radiance data so that samples taken when the hatch is not open can be removed from any subsequent analysis. However, the hatch controller suffered a multi-year failure for the AERI located at the ARM North Slope of Alaska (NSA) Central Facility in Barrow, Alaska, from July 2006-February 2008. The failure resulted in misreporting the state of the hatch in the 'hatchOpen' field within the AERI data files. With this error there is no simple solution to translate what was reported back to the correct hatch status, thereby making it difficult for an analysis to determine when the AERI was actually viewing the sky. As only the data collected when the hatch is fully open are scientifically useful, an algorithm was developed to determine whether the hatch was open or closed based on spectral radiance data from the AERI. Determining if the hatch is open or closed in a scene with low clouds is non-trivial, as low opaque clouds may look very similar spectrally as the closed hatch. This algorithm used a backpropagation neural network; these types of neural networks have been used with increasing frequency in atmospheric science applications.

47 OTHER INSTRUMENTATION↗

Oak Ridge National Laboratory Annual Sustainability Report 2023

ORNL, managed under contract by UT-Battelle LLC, is DOE’s largest science and energy laboratory and, as such, executes the widest range of mission capabilities. Diverse expertise spans a broad range of scientific and engineering disciplines, enabling research and science achievements to accelerate the delivery of solutions to the marketplace. ORNL supports DOE’s national missions of scientific discovery, clean energy, and security. To execute these activities, ORNL has grown significantly over 80 years of continuous operations, consisting of facilities with commissioning dates ranging from the 1940s to the present—an extraordinary set of distinctive scientific facilities and equipment. The complexities of such a variety of facilities require teamwork among divisions, a wide variety of conservation projects, and creative strategies to achieve the desired energy and water savings. Such a diverse and unique set of major facilities, totaling over 5.5 million square feet, with 6,000 employees, requires an innovative plan to accomplish advancements in operational efficiencies. ORNL is tasked with the management of an extraordinary set of distinctive scientific facilities and equipment for DOE. ORNL is mission-driven, and its mission has grown substantially over the decades. ORNL’s core research capabilities provide broad science and technology support for DOE in the areas of energy, environment, and national security. Currently, ORNL is a world leader in materials, neutron, and nuclear science and engineering, and in high-performance computing and data analytics. ORNL’s vast portfolio of research facilities must be maintained and carefully upgraded to protect the nation’s investment in scientific analysis. The goal of sustainable and resilient operations is to enable more effective execution of ORNL’s science and technology mission. Sustainable operational practices and enhanced resilience strive for excellent results while remaining diligent in energy conservation, environmental stewardship, asset management, and community engagement. The Sustainable ORNL Program (Sustainable ORNL) Continuous improvements in operational and business processes must be integrated into the fabric of the ORNL culture to maximize the return from the investment made in modernizing facilities and equipment. The Sustainable ORNL program promotes the legacy of system-wide best practices, management commitment, and employee engagement that will lead ORNL into a future of efficient, resilient, and sustainable operations. ORNL leadership and Sustainable ORNL champions receive regular status reports on the progress of each project and focus area (i.e., roadmap) and periodic summary reports. More information can be found at the program’s website. The Sustainable ORNL roadmap structure endorses 15 vital roadmaps. The figure below summarizes the current project assignments and demonstrates that each project contributes to the wellbeing of the whole. Continuous employee engagement and regular status reports confirm the ideals of the program. The roadmap structure is not static; as the science mission advances and the needs of the organization evolve, the Sustainable ORNL roadmap structure elements are modified to align with developing priorities. In 2022, Sustainable ORNL made roadmap changes to better align ORNL to support new federal requirements that have been issued.

54 ENVIRONMENTAL SCIENCES↗

Robust Dark Energy Constraints with the Dark Energy Spectroscopic Survey (Final Technical Report)

This project developed and applied advanced theoretical, computational, and data-analysis methodologies to extract robust and precise cosmological constraints from the Dark Energy Spectroscopic Instrument (DESI). The work focused on maximizing the scientific return of DESI through optimized survey strategy, novel higher-order clustering statistics, improved modeling of small-scale structure, and rigorous mitigation of observational systematics. Over the award period, the project made substantial contributions to DESI science planning, produced new methods for bispectrum and three-point correlation function analyses, advanced constraints on primordial non-Gaussianity, and delivered widely used software tools. The project also played a major role in training graduate students and a postdoctoral researcher who contributed directly to DESI key projects. The results have significantly enhanced the cosmological reach of DESI and provide a strong foundation for future surveys such as DESI-II and Stage-V experiments.

79 ASTRONOMY AND ASTROPHYSICS↗

Report of the Terrestrial Bodies Science Working Group. Volume 1: Executive summary

Current knowledge of Mercury, Venus, Mars, the Moon, asteroids, comets, and the Galilean satellites were reviewed along with related NASA programs and available mission concepts. Exploration plans for the 1980 to 1990 period are outlined and recommendations made. Topics discussed include: scientific objectives and goals, exploration strategy and recommended mission plans, supporting research and technology, Earth-based and Earth-orbital investigations, data analysis and synthesis, analysis of extraterrestrial materials, broadening the science support base, and international cooperation.

Source record↗

Principal investigator in a box: Version 1.2 documentation

Principal Investigator (PI) in a box is a computer system designed to help optimize the scientific results of experiments that are performed in space. The system will assist the astronaut experimenters in the collection and analysis of experimental data, recognition and pursuit of 'interesting' results, optimal use of the time allocated to the experiment, and troubleshooting of the experiment apparatus. This document discusses the problems that motivate development of 'PI-in-a-box', and presents a high- level system overview and a detailed description of each of the modules that comprise the current version of the system.

Jurine Adolph↗