Gold Standard for macromolecular crystallography diffraction data
Explore the source record for details and available documents.
99 GENERAL AND MISCELLANEOUS↗
SEARCH · Engineering Papers
Results for “data”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Scalable, In-situ Data Clustering Data Analysis for Extreme Scale Scientific Computing (Final Report)
The objective of this project is to address challenges in the design and development of scalable in-situ data clustering and analytics algorithms and software. Our goal is to develop parallel software consisting of a set of spatio-temporal data clustering and anomaly detection functions, both of which are very important for large-scale analysis and have wide applicability for in-situ runs as well as post-processing analysis. Our design principles for in-situ analysis consider the following: (1) identify parts of the computation can be done close to the data within the nodes, while it is still in memory; (2) extract analysis components can (and should) be performed in remote staging and analysis nodes; (3) develop error-bound approximation methods for applications tolerable for small errors; (4) identify the type of derived distributions and statistics, for spatio-temporal data, that can be kept locally in order to both accelerate computations and meet energy constraints in subsequent iterations and phases; (5) use a self-describing data format so that data can be consistent and understood among local storage (memory and SSDs) and at staging and analysis nodes, thereby providing portability and flexibility; (6) develop service-oriented functions that can schedule in-situ and post-hoc analysis tasks based on the dynamic requirements of applications. Our development focus is to produce the parallel data analysis software/library that will be scalable, reusable, extensible, and generic for applications in different disciplines. The software will be able to run in-situ with the simulations as well as post-hoc analysis. This approach will satisfy many synergistic requirements for data intensive applications executed on data coming from instruments and experiments. In particular, the proposed multilevel approach is directly applicable to perform design tradeoffs for running part of the algorithms near the instruments and the rest on remote (analysis) systems.
97 MATHEMATICS AND COMPUTING↗
The Geothermal Data Repository: Ten Years of Supporting the Geothermal Industry with Open Access to Geothermal Data: Preprint
The Department of Energy's (DOE) Geothermal Data Repository (GDR) is celebrating its tenth anniversary! Over the last decade it has grown from the simple idea of storing public data in a centralized location to a valuable tool at the center of the US geothermal scientific community and an integral part of the DOE Geothermal Technologies Office (DOE GTO) project management strategy. Researchers funded by the DOE GTO have contributed over 1,300 data submissions to the GDR. These data have been used to further advancements in geothermal science, economic analysis, exploration, research, development, and operational efficiency. The adoption of open data methodologies and a data management strategy that prioritizes universal open access and standardized, interoperable data have further increased the value of GDR data, making them available across a distributed network of data sharing partners and improving their utility to other industries and related fields, including material science and space exploration. Incorporating feedback from users has been critical to the GDRs success, allowing it to grow over the years to meet the evolving needs of the geothermal community. This paper will explore some of many changes that occurred throughout the GDRs tenure and the lessons learned along the way, as well as highlight some of the new features and recent improvements that been implemented to support innovation, reduce duplication of effort, and advance the geothermal industry as a whole.
accessibility↗
The Geothermal Data Repository: Ten Years of Supporting the Geothermal Industry with Open Access to Geothermal Data
The Department of Energy's (DOE) Geothermal Data Repository (GDR) is celebrating its tenth anniversary! Over the last decade it has grown from the simple idea of storing public data in a centralized location to a valuable tool at the center of the US geothermal scientific community and an integral part of the DOE Geothermal Technologies Office (DOE GTO) project management strategy. Researchers funded by the DOE GTO have contributed over 1,300 data submissions to the GDR. These data have been used to further advancements in geothermal science, economic analysis, exploration, research, development, and operational efficiency. The adoption of open data methodologies and a data management strategy that prioritizes universal open access and standardized, interoperable data have further increased the value of GDR data, making them available across a distributed network of data sharing partners and improving their utility to other industries and related fields, including material science and space exploration. Incorporating feedback from users has been critical to the GDR's success, allowing it to grow over the years to meet the evolving needs of the geothermal community. This paper will explore some of many changes that occurred throughout the GDRs tenure and the lessons learned along the way, as well as highlight some of the new features and recent improvements that been implemented to support innovation, reduce duplication of effort, and advance the geothermal industry as a whole.
access↗
Data Efficiency Assessment of Generative Adversarial Networks for Critical Heat Flux Synthetic Data Generation
This study investigates the application of generative artificial intelligence techniques, particularly conditional generative adversarial networks (cGAN), in real-world engineering contexts, with a specific focus on synthetic data generation for critical heat flux (CHF). Utilizing a dataset comprising more than 20,000 real experimental CHF measurements, we conduct a series of experiments to examine cGAN’s behavior. These experiments encompass varying sizes of the training dataset, training cGAN on data from diverse experimental sources to generate new data on unseen experimental setups, and assessing the impact of excluding various input features on cGAN’s data generation accuracy. Our findings underscore the pronounced data dependency of cGAN for reliable performance, with decreased efficacy observed with smaller training dataset sizes. Notably, cGAN exhibits varying performance when trained on data from different experiments, with superior predictive capabilities observed for certain experiment sources compared to others. For instance, when cGAN was trained on data from Smolin et al.’s experiments or Zenkevich et al., it exhibited relatively good performance in generating the data from Becker et al., Kirillov et al., and Alekseev et al. experiments. In contrast, when trained with Alekseev et al.’s data and tasked with generating other experimental setups, cGAN showed notably poor performance. In both scenarios, cGAN’s performance was inferior compared to training on samples from all experiments concurrently. A feature importance analysis highlights the significant influence of parameters such as mass flux and heated length on accurate CHF generation, while other parameters like diameter and pressure have less impact. Inlet temperature is identified as a moderating factor by cGAN.
22 - GENERAL STUDIES OF NUCLEAR REACTORS↗
Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing
Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.
Choi, Jong Youl↗
Trust Not Verify? The Critical Need for Data Curation Standards in Materials Informatics
The importance of data curation has been recognized in multiple areas of research; however, the discussion of this important issue is only beginning to emerge in materials science. In this Perspective, we highlight the benefits of using the standardized data curation protocols in materials science and discuss current gaps in accurate and reproducible data reporting using case studies drawn from high-impact materials science papers and well-known databases such as the Crystallography Open Database (COD) and the Cambridge Structural Database (CSD). We argue that both experimental and computational materials scientists need to embrace a culture of rigorous data curation as part of modern research data management. We propose a sample data curation pipeline for materials chemistry and illustrate its use by creating two new materials chemistry databases. Here, we hope that this perspective will serve to catalyze further discussion and promote the continuous development of rigorous data curation practices within the materials science research community. We posit that adherence to best practices of data curation will promote and enhance the reliability, reproducibility, and integrity of materials research and enable the development of reliable AI and machine learning models that critically depend on the use of quality data.
Chemical structure↗
Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing
The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS and ALICE are the largest collaborations ever assembled in the sciences and are at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, both experiments rely on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System (WMS) for managing the workflow for all data processing on hundreds of data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. The scale is demonstrated by the following numbers: PanDA manages O(10 2 ) sites, O(10 5 ) cores, O(10 8 ) jobs per year, O(10 3 ) users, and ATLAS data volume is O(10 17 ) bytes. In 2013 we started an ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF). The project titled 'Next Generation Workload Management and Analysis System for Big Data' (BigPanDA) is funded by DOE ASCR and HEP. Extending PanDA to clouds and LCF presents new challenges in managing heterogeneity and supporting workflow. The BigPanDA project is underway to setup and tailor PanDA at the Oak Ridge Leadership Computing Facility (OLCF) and at the National Research Center "Kurchatov Institute" together with ALICE distributed computing and ORNL computing professionals. Our approach to integration of HPC platforms at the OLCF and elsewhere is to reuse, as much as possible, existing components of the PanDA system. Finally, we will present our current accomplishments with running the PanDA WMS at OLCF and other supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications.
97 MATHEMATICS AND COMPUTING↗
Integrating proteomic or transcriptomic data into metabolic models using linear bound flux balance analysis
Abstract Motivation Transcriptomics and proteomics data have been integrated into constraint-based models to influence flux predictions. However, it has been reported recently for Escherichia coli and Saccharomyces cerevisiae, that model predictions from parsimonious flux balance analysis (pFBA), which does not use expression data, are as good or better than predictions from various algorithms that integrate transcriptomics or proteomics data into constraint-based models. Results In this paper, we describe a novel constraint-based method called Linear Bound Flux Balance Analysis (LBFBA), which uses expression data (either transcriptomic or proteomic) to predict metabolic fluxes. The method uses expression data to place soft constraints on individual fluxes, which can be violated. Parameters in the soft constraints are first estimated from a training expression and flux dataset before being used to predict fluxes from expression data in other conditions. We applied LBFBA to E.coli and S.cerevisiae datasets and found that LBFBA predictions were more accurate than pFBA predictions, with average normalized errors roughly half of those from pFBA. For the first time, we demonstrate a computational method that integrates expression data into constraint-based models and improves quantitative flux predictions over pFBA. Availability and implementation Code is available in the Supplementary data available at Bioinformatics online. Supplementary information Supplementary data are available at Bioinformatics online.
Tian, Mingyuan↗
Galaxy bias from the Dark Energy Survey Science Verification data: Combining galaxy density maps and weak lensing maps
We measure the redshift evolution of galaxy bias for a magnitude-limited galaxy sample by combining the galaxy density maps and weak lensing shear maps for a ~116 deg 2 area of the Dark Energy Survey (DES) Science Verification (SV) data. This method was first developed in Amara et al. and later re-examined in a companion paper with rigorous simulation tests and analytical treatment of tomographic measurements. In this work we apply this method to the DES SV data and measure the galaxy bias for a i < 22.5 galaxy sample. We find the galaxy bias and 1σ error bars in four photometric redshift bins to be 1.12 ± 0.19 (z = 0.2–0.4), 0.97 ± 0.15 (z = 0.4–0.6), 1.38 ± 0.39 (z = 0.6–0.8), and 1.45 ± 0.56 (z = 0.8–1.0). These measurements are consistent at the 2σ level with measurements on the same data set using galaxy clustering and cross-correlation of galaxies with cosmic microwave background lensing, with most of the redshift bins consistent within the 1σ error bars. In addition, our method provides the only σ8 independent constraint among the three. We forward model the main observational effects using mock galaxy catalogues by including shape noise, photo-z errors, and masking effects. We show that our bias measurement from the data is consistent with that expected from simulations. With the forthcoming full DES data set, we expect this method to provide additional constraints on the galaxy bias measurement from more traditional methods. Moreover, in the process of our measurement, we build up a 3D mass map that allows further exploration of the dark matter distribution and its relation to galaxy evolution.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
A revised partiality model and post-refinement algorithm for X-ray free-electron laser data
An updated partiality model and post-refinement algorithm for XFEL snapshot diffraction data is presented and confirmed by observing anomalous density for S atoms at an X-ray wavelength of 1.3 Å. Research towards using X-ray free-electron laser (XFEL) data to solve structures using experimental phasing methods such as sulfur single-wavelength anomalous dispersion (SAD) has been hampered by shortcomings in the diffraction models for X-ray diffraction from FELs. Owing to errors in the orientation matrix and overly simple partiality models, researchers have required large numbers of images to converge to reliable estimates for the structure-factor amplitudes, which may not be feasible for all biological systems. Here, data for cytoplasmic polyhedrosis virus type 17 (CPV17) collected at 1.3 Å wavelength at the Linac Coherent Light Source (LCLS) are revisited. A previously published definition of a partiality model for reflections illuminated by self-amplified spontaneous emission (SASE) pulses is built upon, which defines a fraction between 0 and 1 based on the intersection of a reflection with a spread of Ewald spheres modelled by a super-Gaussian wavelength distribution in the X-ray beam. A method of post-refinement to refine the parameters of this model is suggested. This has generated a merged data set with an overall discrepancy (by calculating the R{sub split} value) of 3.15% to 1.46 Å resolution from a 7225-image data set. The atomic numbers of C, N and O atoms in the structure are distinguishable in the electron-density map. There are 13 S atoms within the 237 residues of CPV17, excluding the initial disordered methionine. These only possess 0.42 anomalous scattering electrons each at 1.3 Å wavelength, but the 12 that have single predominant positions are easily detectable in the anomalous difference Fourier map. It is hoped that these improvements will lead towards XFEL experimental phase determination and structure determination by sulfur SAD and will generally increase the utility of the method for difficult cases.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.
Godoy, William↗
Using Landsat Surface Reflectance Data as a Reference Target for Multiswath Hyperspectral Data Collected Over Mixed Agricultural Rangeland Areas
Low-cost flight-based hyperspectral imaging systems have the potential to provide important information for ecosystem and environmental studies as well as aide in land management. To realize this potential, methods must be developed to provide large-area surface reflectance data allowing for temporal data sets at the mesoscale. This paper describes a bootstrap method of producing a large-area, radiometrically referenced hyperspectral data set using the Landsat surface reflectance (LaSRC) data product as a reference target. The bootstrap method uses standard hyperspectral processing techniques that are extended to remove uneven illumination conditions between flight passes, allowing for radiometrically self-consistent data after mosaicking. Through selective spectral and spatial resampling, LaSRC data are used as a radiometric reference target. Advantages of the bootstrap method include the need for minimal site access, no ancillary instrumentation, and automated data processing. Data from two hyperspectral flights over the same managed agricultural and unmanaged range land covering approximately 5.8 km 2 acquired on June 21, 2014 and June 24, 2015 are presented. As a result, data from a flight over agricultural land collected on June 6, 2016 are compared with concurrently collected ground-based reflectance spectra as a means of validation.
54 ENVIRONMENTAL SCIENCES↗
Joint physics-based and data-driven time-lapse seismic inversion: Mitigating data scarcity
In carbon capture and sequestration (CCS), developing rapid and effective imaging techniques is crucial for real-time monitoring of the spatial and temporal dynamics of CO 2 propagation during/after injection. With continuing improvements in computational power and data storage, data-driven techniques based on machine learning (ML) have been effectively applied to seismic inverse problems. In particular, ML helps alleviate the ill-posedness and high computational cost of full-waveform inversion (FWI). However, such data-driven inversion techniques require massive high-quality training data sets to ensure prediction accuracy, which hinders their application to time-lapse monitoring of CO 2 sequestration. We propose an efficient “hybrid” time-lapse workflow that combines physics-based FWI and data-driven ML inversion. The scarcity of the available training data is addressed by developing a new data-generation technique with physics constraints. The method is vali dated on a synthetic CO 2 -sequestration model based on the Kimberlina storage reservoir in California. The proposed approach is shown to synthesize a large volume of high-quality, physically realistic training data, which is critically important in accurately characterizing the CO 2 movement in the reservoir. In conclusion, the developed hybrid methodology can also simultaneously predict the variations in velocity and saturation and achieve high spatial resolution in the presence of realistic noise in the data.
58 GEOSCIENCES↗
Fine-Root Ecology Database (FRED): A Global Collection of Root Trait Data with Coincident Site, Vegetation, Edaphic, and Climatic Data, Version 4.
To address the need for a centralized root trait database, we compiled the Fine-Root Ecology Database (FRED) from published and unpublished data sources. We have continued to add to the FRED database since the release of FRED 1.0 in 2017, followed by 2.0 in 2018, and 3.0 in 2021. This new release of FRED 4.0 now has 213,941 observations of 238 root traits, for a combined total of roughly 3.4 million data fields for root traits and ancillary data together. FRED 4.0 has 39.8% more root trait observations than FRED 3.0 and a 34.4% increase in unique data sources. This release of FRED 4.0 also includes significant increases in geographic regions that have long been underrepresented in global datasets, notably in the tropical low latitudes. Ancillary data on associated site, vegetation, edaphic, and climatic conditions from across the globe have also increased concurrently with root trait observations. FRED is focused on fine roots (traditionally defined as roots less than 2 mm in diameter), as coarse roots are studied using different methodology, often at very different scales, and have different traits and trait interpretations. Despite this fine-root focus, FRED accepts data collected from roots of all sizes and contains observations of many root classes including coarse roots. Data collection will continue for the foreseeable future. The FRED4_Entire_Database_2026.csv file is the flat csv data file for FRED 4.0, and the FRED4_dd.csv file is the data dictionary of all columns available in FRED, including column IDs, column names, definitions, and unit (where applicable).
54 ENVIRONMENTAL SCIENCES↗
Contribution to the Development of DOE ARM Climate Best Estimate Data (ARMBE) Products: Land Surface Forcing and Evaluation Data Set (ARMBE-LAND) from the ARM SGP Site
To support the LLNL ARM Best Estimate Data Products (ARMBE) data development, the San Jose State University (SJSU)’s group will provide the LLNL team yearly data files (in a format consistent with ARMBE) that contain variables from the instruments defined in Table 1 for the ARM SGP site during the available period. The ARMBE-Land data product will blend multiple ARM observations including the Energy Balance Bowen Ratio Station (EBBR), Soil Water and Temperature System (SWATs) and CO2 Flux (CO2FLX) data sets. The SJSU group will process these datasets with stringent quality controls to remove suspicious data points and average quality-controlled data over a desired temporal resolution (one hour). A qc-flag will be assigned to each variable to indicate how many valid data points are used to get the hourly mean data, consistent with other ARMBE fileds.
54 ENVIRONMENTAL SCIENCES↗
ATDM Data Management FY2015: Data Warehouse Progress Report
The Advanced Technology Development and Mitigation (ATDM) program at Sandia National Laboratories is a new effort to build next-generation simulation codes that will map well to upcoming exascale computing platforms. Rather than follow traditional single- program, multiple data (SPMD) programming techniques, ATDM is developing applications in an asynchronous many task (AMT) form that describes work as a graph of tasks that have data dependencies. The data management team is focused on developing a data warehouse for ATDM that will enable tasks to store and exchange data objects efficiently. This report summarizes the data management teams efforts during FY15, and documents: (1) an initial API and implementation for the data warehouses key/value store, (2) API requirements for use with ATDMs runtime, (3) initial requirements for storing ATDM-specific data, and (4) the current organization of software components that will be used by the data warehouse.
97 MATHEMATICS AND COMPUTING↗
Dynamically Reconfigurable Data Readout of Pixel Detectors for Automatic Synchronization with Data Acquisition Systems
Reconfigurable detectors with dynamically selectable sensing and readout modes are highly desirable for implementing edge computing as well as enabling advanced imaging techniques such as foveation. The concept of a camera system capable of simultaneous passive imaging and dynamic ranging in different regions of the detector is presented. Such an adaptive-autonomous detector with both spatial and temporal control requires programmable window of exposure (time frames), ability to switch between readout modes such as full-frame imaging and zero-suppressed data, modification of the number of pixel data bits and independent programmability for distinct detector regions. In this work, a method is presented for seamlessly changing time frames and readout modes without data corruption while still ensuring that the data acquisition system (DAQ) does not need to stop and resynchronize at each change of setting, thus avoiding significant dead time. Data throughput is maximized by using a minimum unique data format, rather than lengthy frame headers, to differentiate between consecutive frames. A data control and transmitter (DCT) synchronizes data transfer from the pixel to the periphery, reconfigures the data to transmit it serially off-chip, while providing optimized decision support based on a DAQ definable mode. Measurements on a test structure demonstrate that the DCT can operate at 1 GHz in a 65 nm LP CMOS process.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗