Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Scientific Data Management”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

DOE Data Day 2019 Report

The Department of Energy (DOE) has joined the larger scientific community in the promotion of data management as a means to higher quality, more efficient research and analysis, and as a critical component of data science. Tools and platforms to support data management and analysis are rapidly evolving and provide enormous opportunities. They also pose challenges that can be specific to DOE but are common across DOE mission areas and organizations. The DOE Data Day (D3) workshop was held on September 25–26, 2019, and organized by a team at Lawrence Livermore National Laboratory (LLNL) in an effort to gather the data management practitioners at the DOE labs together to share their work and results, facilitating knowledge transfers and best practices across project teams. Over 100 attendees participated from multiple national labs—LLNL, Sandia (SNL), Lawrence Berkeley (LBNL), Los Alamos (LANL), Oak Ridge (ORNL), Argonne (ANL), Pacific Northwest (PNNL), Stanford Linear Accelerator (SLAC)—along with Mission Support and Test Services (MSTS), National Energy Technology Laboratory (NETL), Pantex, National Security Complex (Y-12), National Nuclear Security Administration (NNSA), the DOE Office of Scientific and Technical Information (OSTI), and elsewhere in the DOE complex.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Evaluation of a scientific data search infrastructure

The ability to search over large scientific datasets has become crucial to next-generation scientific discoveries as data generated from scientific facilities grow dramatically. In previous work, we developed and deployed ScienceSearch, a search infrastructure for scientific data which uses machine learning to automate metadata creation. Our current deployment is deployed atop a container based platform at a HPC center. In this article, we present an evaluation and discuss our experiences with the ScienceSearch infrastructure. Specifically, we present a performance evaluation of ScienceSearch's infrastructure focusing on scalability trends. The obtained results show that ScienceSearch is able to serve up to 130 queries/min with latency under 3 s. We discuss our infrastructure setup and evaluation results to provide our experiences and a perspective on opportunities and challenges of our search infrastructure.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Managing a Cray supercomputer as a git branch

Summary Modern software engineering/DevOps techniques applied to Cray XC Series systems management enable higher fidelity translation from test to production environments, reduce administration costs by avoiding duplicate efforts, and increase reliability. This can be done by using a git repository to track and manage all system configurations on the SMW(s) (System Management Workstation) and then adapt a gitflow‐like development methodology. Using git improves change management with peer review of changes, identification of who made a change, knowing why a change was made, facilitating easier reversion of bad changes, and enables a workflow using the ideas from gitflow with individual branches for preparing, maintaining, and recording releases. Using branches enables multiple contributors to work on different types of changes without getting unexpected changes from another feature area. We will describe a process similar to that used at NERSC (National Energy Research Scientific Computing Center) to extract/abstract configuration data, manage it using git, and then re‐apply changes to the SMW(s).

Jacobsen, Douglas M.↗

Nationwide Buildings Energy Research enabled through an integrated Data Intensive Scientific Workflow and Advanced Analysis Environment

Modern workflow systems enable scientists to run ensemble simulations at unprecedented scales and levels of complexity, allowing them to study system sizes previously impossible to achieve, due to the inherent resource requirements needed for the modeling work. However as a result of these new capabilities the science teams suddenly also face unprecedented data volumes that they are unable to analyze with their existing tools and methodologies in a timely fashion. In this paper we will describe the ongoing development work to create an integrated data intensive scientific workflow and analysis environment that offers researchers the ability to easily create and execute complex simulation studies and provides them with different scalable methods to analyze the resulting data volumes. The integration of simulation and analysis environments is hereby not only a question of ease of use, but supports fundamental functions in the correlated analysis of simulation input, execution details and derived results for multi-variant, complex studies. To this end the team extended and integrated the existing capabilities of the Velo data management and analysis infrastructure, the MeDICi data intensive workflow system and RHIPE the R for Hadoop version of the well-known statistics package, as well as developing a new visual analytics interface for the result exploitation by multi-domain users. The capabilities of the new environment are demonstrated on a use case that focusses on the Pacific Northwest National Laboratory (PNNL) building energy team, showing how they were able to take their previously local scale simulations to a nationwide level by utilizing data intensive computing techniques not only for their modeling work, but also for the subsequent analysis of their modeling results. As part of the PNNL research initiative PRIMA (Platform for Regional Integrated Modeling and Analysis) the team performed an initial 3 year study of building energy demands for the US Eastern Interconnect domain, which they are now planning to extend to predict the demand for the complete century. The initial study raised their data demands from a few GBs to 400GB for the 3year study and expected tens of TBs for the full century.

99 GENERAL AND MISCELLANEOUS↗

EOSDIS - Its role in the EOS program and its importance to the scientific community

In order to manage the global data sets required to understand the earth as a system, the EOS Data and Information System (EOSDIS) will collect and store satellite, aircraft, and in situ measurements and their resultant data products, and will distribute the data conveniently. EOSDIS will also provide product generation and science computing facilities to support the development, processing, and validation of standard EOS science data products. The overall architecture of EOSDIS, and how the Distributed Active Archive Centers fit into that structure, are shown. EOSDIS will enable users to query data bases nationally, make use of keywords and other mnemonic identifiers, and see graphic images of subsets of available data prior to ordering full (or selected pieces of) data sets for use in their 'home' environment.

Price, Robert D.↗

A rapid prototyping/artificial intelligence approach to space station-era information management and access

Applications of rapid prototyping and Artificial Intelligence techniques to problems associated with Space Station-era information management systems are described. In particular, the work is centered on issues related to: (1) intelligent man-machine interfaces applied to scientific data user support, and (2) the requirement that intelligent information management systems (IIMS) be able to efficiently process metadata updates concerning types of data handled. The advanced IIMS represents functional capabilities driven almost entirely by the needs of potential users. Space Station-era scientific data projected to be generated is likely to be significantly greater than data currently processed and analyzed. Information about scientific data must be presented clearly, concisely, and with support features to allow users at all levels of expertise efficient and cost-effective data access. Additionally, mechanisms for allowing more efficient IIMS metadata update processes must be addressed. The work reported covers the following IIMS design aspects: IIMS data and metadata modeling, including the automatic updating of IIMS-contained metadata, IIMS user-system interface considerations, including significant problems associated with remote access, user profiles, and on-line tutorial capabilities, and development of an IIMS query and browse facility, including the capability to deal with spatial information. A working prototype has been developed and is being enhanced.

Carnahan, Richard S., Jr.↗

Solid waste management challenges for cities in developing countries

Highlights: Black-Right-Pointing-Pointer Stakeholders. Black-Right-Pointing-Pointer Factors affecting performance waste management systems. Black-Right-Pointing-Pointer Questionnaire as Annex for waste management baseline assessment. - Abstract: Solid waste management is a challenge for the cities' authorities in developing countries mainly due to the increasing generation of waste, the burden posed on the municipal budget as a result of the high costs associated to its management, the lack of understanding over a diversity of factors that affect the different stages of waste management and linkages necessary to enable the entire handling system functioning. An analysis of literature on the work done and reported mainly in publications from 2005 to 2011, related to waste management in developing countries, showed that few articles give quantitative information. The analysis was conducted in two of the major scientific journals, Waste Management Journal and Waste Management and Research. The objective of this research was to determine the stakeholders' action/behavior that have a role in the waste management process and to analyze influential factors on the system, in more than thirty urban areas in 22 developing countries in 4 continents. A combination of methods was used in this study in order to assess the stakeholders and the factors influencing the performance of waste management in the cities. Data was collected from scientific literature, existing data bases, observations made during visits to urban areas, structured interviews with relevant professionals, exercises provided to participants in workshops and a questionnaire applied to stakeholders. Descriptive and inferential statistic methods were used to draw conclusions. The outcomes of the research are a comprehensive list of stakeholders that are relevant in the waste management systems and a set of factors that reveal the most important causes for the systems' failure. The information provided is very useful when planning, changing or implementing waste management systems in cities.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Improving Data Mobility & Management for International Cosmology

In February 2015 the third workshop in the CrossConnects series, with a focus on Improving Data Mobility & Management for International Cosmology, was held at Lawrence Berkeley National Laboratory. Scientists from fields including astrophysics, cosmology, and astronomy collaborated with experts in computing and networking to outline strategic opportunities for enhancing scientific productivity and effectively managing the ever-increasing scale of scientific data. While each field has unique details which depend on the instruments employed, the type and scale of the data, and the structure of scientific collaborations, several important themes emerged from the workshop discussions. Findings, as well as a set of recommendations, are contained in their respective sections in this report.

79 ASTRONOMY AND ASTROPHYSICS↗

BEAM: A Computational Workflow System for Managing and Modeling Material Characterization Data in HPC Environments

Improvements in scientific instrumentation allow imaging at mesoscopic to atomic length scales, many spectroscopic modes, and now—with the rise of multimodal acquisition systems and the associated processing capability—the era of multidimensional, informationally dense data sets has arrived. Technical issues in these combinatorial scientific fields are exacerbated by computational challenges best summarized as a necessity for drastic improvement in the capability to transfer, store, and analyze large volumes of data. The Bellerophon Environment for Analysis of Materials (BEAM) platform provides material scientists the capability to directly leverage the integrated computational and analytical power of High Performance Computing (HPC) to perform scalable data analysis and simulation via an intuitive, cross-platform client user interface. This framework delivers authenticated, “push-button” execution of complex user workflows that deploy data analysis algorithms and computational simulations utilizing the converged compute-and-data infrastructure at Oak Ridge National Laboratory's (ORNL) Compute and Data Environment for Science (CADES) and HPC environments like Titan at the Oak Ridge Leadership Computing Facility (OLCF). In this work we address the underlying HPC needs for characterization in the material science community, elaborate how BEAM's design and infrastructure tackle those needs, and present a small sub-set of user cases where scientists utilized BEAM across a broad range of analytical techniques and analysis modes.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

The Importance of User Feedback in Sustaining Trusted Repositories

The NASA Earth Observation System Data and Information System (EOSDIS) has been operating since 1994 and is serving a global user community with well-managed Earth science data in a variety of scientific disciplines. EOSDIS processes, archives and distributes data and information products resulting from spaceborne and airborne instruments as well as in situ measurements from field campaigns. The Earth Science Data and Information System (ESDIS) Project at the NASA Goddard Space Flight Center manages EOSDIS with its 12 Distributed Active Archive Centers (DAACs) located across the United States. During the entire life of EOSDIS, the ESDIS Project and the DAACs have deployed many different mechanisms for user feedback, which have proven extremely valuable to their evolution and performance in their service to user communities. Some of the inputs from our user groups have resulted in fundamental changes in the architecture, design and operations of EOSDIS, while others have provided novel ideas for incremental changes. The EOSDIS DAACs have User Working Groups (UWGs) that represent broad user communities in the Earth science disciplines served by the DAACs. The UWGs meet periodically to assess and provide feedback on dataset and service priorities. As regular users of the data and services of the DAAC and experts in the scientific disciplines, the UWG members provide valuable inputs for planning and prioritizing the services, as well as addition of new datasets, for the benefit of the community. The EOSDIS is evaluated annually through an independently administered survey of its users resulting in the American Customer Satisfaction Index (ACSI). The survey provides an ACSI score as well as free-text suggestions from users, which are also helpful in making specific system improvements. In addition, each of the DAACs has a user services group that address on-going requests for help and other comments from users. The ESDIS Project has established a mechanism through the "earthdata" website (http://earthdata.nasa.gov) for users to provide feedback of any kind and these questions/comments are routed to the appropriate individuals in the Project or the DAACs. These various forms of receiving user feedback and responding to them continue to be extremely valuable in evolving and sustaining our Earth Science repository.

Behnke, Jeanne↗

The NGEE Arctic Data Archive -- Portal for Archiving and Distributing Data and Documentation

The Next-Generation Ecosystem Experiments (NGEE Arctic) project is committed to implementing a rigorous and high-quality data management program. The goal is to implement innovative and cost-effective guidelines and tools for collecting, archiving, and sharing data within the project, the larger scientific community, and the public. The NGEE Arctic web site is the framework for implementing these data management and data sharing tools. The open sharing of NGEE Arctic data among project researchers, the broader scientific community, and the public is critical to meeting the scientific goals and objectives of the NGEE Arctic project and critical to advancing the mission of the Department of Energy (DOE), Office of Science, Biological and Environmental (BER) Terrestrial Ecosystem Science (TES) program.

Boden, Thomas A.↗

Linking Automated Data Analysis and Visualization with Applications in Developmental Biology and High-Energy Physics

Knowledge discovery from large and complex collections of today's scientific datasets is a challenging task. With the ability to measure and simulate more processes at increasingly finer spatial and temporal scales, the increasing number of data dimensions and data objects is presenting tremendous challenges for data analysis and effective data exploration methods and tools. Researchers are overwhelmed with data and standard tools are often insufficient to enable effective data analysis and knowledge discovery. The main objective of this thesis is to provide important new capabilities to accelerate scientific knowledge discovery form large, complex, and multivariate scientific data. The research covered in this thesis addresses these scientific challenges using a combination of scientific visualization, information visualization, automated data analysis, and other enabling technologies, such as efficient data management. The effectiveness of the proposed analysis methods is demonstrated via applications in two distinct scientific research fields, namely developmental biology and high-energy physics.Advances in microscopy, image analysis, and embryo registration enable for the first time measurement of gene expression at cellular resolution for entire organisms. Analysis of high-dimensional spatial gene expression datasets is a challenging task. By integrating data clustering and visualization, analysis of complex, time-varying, spatial gene expression patterns and their formation becomes possible. The analysis framework MATLAB and the visualization have been integrated, making advanced analysis tools accessible to biologist and enabling bioinformatic researchers to directly integrate their analysis with the visualization. Laser wakefield particle accelerators (LWFAs) promise to be a new compact source of high-energy particles and radiation, with wide applications ranging from medicine to physics. To gain insight into the complex physical processes of particle acceleration, physicists model LWFAs computationally. The datasets produced by LWFA simulations are (i) extremely large, (ii) of varying spatial and temporal resolution, (iii) heterogeneous, and (iv) high-dimensional, making analysis and knowledge discovery from complex LWFA simulation data a challenging task. To address these challenges this thesis describes the integration of the visualization system VisIt and the state-of-the-art index/query system FastBit, enabling interactive visual exploration of extremely large three-dimensional particle datasets. Researchers are especially interested in beams of high-energy particles formed during the course of a simulation. This thesis describes novel methods for automatic detection and analysis of particle beams enabling a more accurate and efficient data analysis process. By integrating these automated analysis methods with visualization, this research enables more accurate, efficient, and effective analysis of LWFA simulation data than previously possible.

3D Gene Expression↗

Extensible Computational Chemistry Environment

ECCE provides a sophisticated graphical user interface, scientific visualization tools, and the underlying data management framework enabling scientists to efficiently set up calculations and store, retrieve, and analyze the rapidly growing volumes of data produced by computational chemistry studies. ECCE was conceived as part of the Environmental Molecular Sciences Laboratory construction to solve the problem of researchers being able to effectively utilize complex computational chemistry codes and massively parallel high performance compute resources. Bringing the power of these codes and resources to the desktops of researcher and thus enabling world class research without users needing a detailed understanding of the inner workings of either the theoretical codes or the supercomputers needed to run them was a grand challenge problem in the original version of the EMSL. ECCE allows collaboration among researchers using a web-based data repository where the inputs and results for all calculations done within ECCE are organized. ECCE is a first of kind end-to-end problem solving environment for all phases of computational chemistry research: setting up calculations with sophisticated GUI and direct manipulation visualization tools, submitting and monitoring calculations on remote high performance supercomputers without having to be familiar with the details of using these compute resources, and performing results visualization and analysis including creating publication quality images. ECCE is a suite of tightly integrated applications that are employed as the user moves through the modeling process.

Source record↗

COVID-19, An Exercise in Data Governance at Sandia National Laboratories

In April of 2020, Sandia National Laboratories had an urgent need to identify and manage the data that could be used to create mobile applications, models, reports, and visualizations to assist management in safely bringing the workforce onsite during the COVID-19 pandemic. Multiple divisions volunteered to design and build software solutions; meanwhile, requests for new data sources, including duplicate requests, were inundating Information Technology (IT) and data owners. The Enterprise Data Governance Team was assigned to resolve obtaining and accessing new sources of data in an accelerated timeframe. Through successful collaboration with multiple stakeholders and domain owners across Sandia, the Enterprise Data Governance Team rapidly developed a centralized data strategy and solution for use in safeguarding the Sandia workforce during the COVID-19 pandemic. This foundation enabled teams to successfully develop solutions, including reports for executives and management as well as the data for modeling and scientific analysis.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Telemetry packetization for improved mission operations

The requirements for mission-operations data management will accelerate sharply when the Space Transportation System (i.e., Space Shuttle) becomes the primary vehicle for research from space. These demands can be satisfied most effectively by providing a higher-level source encoding function within the spaceborne vehicle. An Instrument Telemetry Packet (ITP) concept is described which represents an alternative to the conventional multiplexed telemetry frame approach for acquiring spaceborne instrument data. By providing excellent data-integrity protection at the source and a variable instrument bandwidth capability, this ITP concept represents a significant improvement over present data acquisition procedures. Realignments in the ground telemetry processing functions are described which are intended to take advantage of the ITP concept and to make the data management system more responsive to the scientific investigators.

Greene, E. P.↗

The preparation of the measurements at the PDS at the Padova Observatory: Report on an experience with the development of an astronomical data base environment

Some software facilities used mainly for information retrieval and analysis at the Padova-Asiago Observatory are discussed. These facilities help guest and resident astronomers to make easier the preparation of plate measurements. The problems connected with the creation, use and management of a data base in a scientific (astronomical) environment are reviewed on the basis of the experience gathered during the last three years. The development plan of the user session environment and its possible applications in a computer network are briefly sketched.

Benacchio, L.↗

Hybrid LZW compression

The Science Data Management and Science Payload Operations subpanel reports from the NASA Conference on Scientific Data Compression (Snowbird, Utah in 1988) indicate the need for both lossless and lossy image data compression systems. The ranges developed by the subpanel suggest ratios of 2:1 to 4:1 for lossless coding and 2:1 to 6:1 for lossy predictive coding. For the NASA Freedom Science Video Processing Facility it would be highly desirable to implement one baseline compression system which would meet both of these criteria. Presented here is such a system, utilizing an LZW hybrid coding scheme which is adaptable to either type of compression. Simulation results are presented with the hybrid LZW algorithm operating in each of its modes.

Lewis, H. Garton, Jr.↗

Earth science data study

The research proposed in this contract concerning investigations of existing and planned Earth Science and Applications Division (ESAD) data management systems and research into utilities for the access and display of scientific data products was completed. A summary of this work is provided.

Graves, Sara J.↗