Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Observation Data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Earth System Grid Center for Enabling Technologies (ESG-CET): A Data Infrastructure for Data-Intensive Climate Research

For the Earth System Grid Federation (ESGF), the ESG-CET team has led international development and delivered a production environment for managing and accessing ultrascale climate data. This production environment includes multiple national and international climate projects (e.g., Couple Model Intercomparison Project, Community Earth System Model), ocean model data (such as the Parallel Ocean Program), observation data (Carbon Dioxide Information and Analysis Center, Atmospheric Infrared Sounder, and so forth), and analysis and visualization tools, all of which serve a diverse community of users. These data holdings and services are distributed across multiple ESG-CET sites (such as LANL, LBNL, LLNL, NCAR, and ORNL) as well as at unfunded partners sites such as the Australian National University National Computational Infrastructure, the British Atmospheric Data Centre, the National Oceanic and Atmospheric Administration Geophysical Fluid Dynamics Laboratory, the Max Planck Institute for Meteorology, the German Climate Computing Centre, and the National Aeronautics and Space Administration Jet Propulsion Laboratory. More recently, ESG-CET has been extending services beyond data-file access and delivery to develop more detailed information products (scientific graphics, animations, etc.), secure binary data-access services (based upon the OPeNDAP protocol), and server-side analysis capabilities. These will allow users to request data subsets transformed through commonly used analysis and intercomparison procedures. As we transition from development activities to production and operations, the ESG-CET team is tasked with making data available to all users seeking to understand, process, extract value from, visualize, and/or communicate it to others. This ongoing effort, though daunting in scope and complexity, will greatly magnify the value of numerical climate model outputs and climate observations for future national and international climate-assessment reports. Continued ESGF progress will result in a production ultrascale data system for empowering scientists who attempt new and exciting data exchanges that could ultimately lead to breakthrough climate-science discoveries.

54 ENVIRONMENTAL SCIENCES↗

Multi-Formalism Modeling for Disaster Resilience, Forecasting, and Response

Disaster response preparedness, risk mitigation, and effective post-event response are critical for national security, public health, and economic integrity. For areas impacted by extreme events, timely and accurate impact assessments of lifeline infrastructures are critical. Traditional disaster management and situational awareness platforms are not capable of handling the massive influx of available information from traditional and non-traditional sources. Next-generation analytic assessment approaches, where use of dynamically sourced, heterogeneous data with near real-time availability are integrated into a continually-updated analytics framework. The objective of this research was to develop a foundational multi-formalism modeling platform to integrate novel data sources for use in improving situational awareness after a disaster event. Specifically, to 1) provide an architecture for machine-learning based data fusion various multi-sensor datasets; 2) classify, geolocate, and temporally bin novel data sources for use in spatially-enabled machine-learning based data fusion; and 3) derive confidence-based observation data with machine-learning based prediction to fill spatial and temporal observation gaps. This project developed a reusable cloud-based pipeline architecture for ingesting, storing, and analyzing relevant data, executing a range of deep-learning models, and visualize the information geographically. The pipeline development was informed by previous PNNL investments (StreamKit) and built upon leading-edge technologies consisting of: 1) a raw datastore; 2) a data enrichment pipeline with custom and third-party analytics; 3) enriched data storage; and 4) a custom UI to display the enriched data. Machine learning/deep learning models were selected and tested to classify social media images for relevancy and presence of flood waters. Our custom trained relevance models were run on a validation dataset and the overall accuracy of the relevance classifiers ranged from 65% to 90%. We found that convolutional neural network (CNN) models were both faster and more accurate than more traditional machine learning models. We applied the relevance classifier to remove non-relevant images and then undertook a limited training of CNN models to detect flood on ~1,800 labeled images. The testing accuracy of the flood detection models were ~88%. To improve the geographic location of both social media and oblique aerial images we tested a multi-step workflow built on a pre-trained deep-learning model. To identify similarity features from test and reference scenes, we applied the Xception and Faiss models to a set of test images with inaccurate spatial locations, as well and precisely-located reference scenes from the general area. Matching reference scenes to the test scenes with the closest similarity index had limited success in improving the precision of test scene location and additional methods have been identified for testing. Data fusion between authoritative geographic data (satellite imaging) and non-authoritative crowdsourced data is an underlying theme throughout this research and methods are tested bring these data to a common geographic form through feature matching and transformations. The automated extraction of supporting information for disaster response from oblique aerial images or ground-level social media images is accomplished through semantic image segmentation and transformation. The non-authoritative data are used to validate the authoritative data collections and processing and optionally, t+0 flood models. In addition, the common form geographic data are combined with other static landscape features to develop spatially-explicit training vectors used in deep generative statistical inference models to develop complex non-linear relationships in static and dynamic data and ultimately derive a spatially continuous and probabilistic assessment of flood damage with the event domain.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Using Data-Driven Prediction of Downstream 1D River Flow to Overcome the Challenges of Hydrologic River Modeling

Methods for downstream river flow prediction can be categorized into physics-based and empirical approaches. Although based on well-studied physical relationships, physics-based models rely on numerous hydrologic variables characteristic of the specific river system that can be costly to acquire. Moreover, simulation is often computationally intensive. Conversely, empirical models require less information about the system being modeled and can capture a system’s interactions based on a smaller set of observed data. This article introduces two empirical methods to predict downstream hydraulic variables based on observed stream data: a linear programming (LP) model, and a convolutional neural network (CNN). We apply both empirical models within the Colorado River system to a site located on the Green River, downstream of the Yampa River confluence and Flaming Gorge Dam, and compare it to the physics-based model Streamflow Synthesis and Reservoir Regulation (SSARR) currently used by federal agencies. Results show that both proposed models significantly outperform the SSARR model. Moreover, the CNN model outperforms the LP model for hourly predictions whereas both perform similarly for daily predictions. Although less accurate than the CNN model at finer temporal resolution, the LP model is ideal for linear water scheduling tools.

13 HYDRO ENERGY↗

Development of a Hydrodynamic and Transport model of Bellingham Bay in Support of Nearshore Habitat Restoration

In this study, a hydrodynamic model based on the unstructured-grid finite volume coastal ocean model (FVCOM) was developed for Bellingham Bay, Washington. The model simulates water surface elevation, velocity, temperature, and salinity in a three-dimensional domain that covers the entire Bellingham Bay and adjacent water bodies, including Lummi Bay, Samish Bay, Padilla Bay, and Rosario Strait. The model was developed using Pacific Northwest National Laboratory’s high-resolution Puget Sound and Northwest Straits circulation and transport model. A sub-model grid for Bellingham Bay and adjacent coastal waters was extracted from the Puget Sound model and refined in Bellingham Bay using bathymetric light detection and ranging (LIDAR) and river channel cross-section data. The model uses tides, river inflows, and meteorological inputs to predict water surface elevations, currents, salinity, and temperature. A tidal open boundary condition was specified using standard National Oceanic and Atmospheric Administration (NOAA) predictions. Temperature and salinity open boundary conditions were specified based on observed data. Meteorological forcing (wind, solar radiation, and net surface heat flux) was obtained from NOAA real observations and National Center for Environmental Prediction North American Regional Analysis outputs. The model was run in parallel with 48 cores using a time step of 2.5 seconds. It took 18 hours of cpu time to complete 26 days of simulation. The model was calibrated with oceanographic field data for the period of 6/1/2009 to 6/26/2009. These data were collected specifically for the purpose of model development and calibration. They include time series of water-surface elevation, currents, temperature, and salinity as well as temperature and salinity profiles during instrument deployment and retrieval. Comparisons between model predictions and field observations show an overall reasonable agreement in both temporal and spatial scales. Comparisons of root mean square error values for surface elevation, velocity, temperature, and salinity time series are 0.11 m, 0.10 m/s, 1.28oC, and 1.91 ppt, respectively. The model was able to reproduce the salinity and temperature stratifications inside Bellingham Bay. Wetting and drying processes in tidal flats in Bellingham Bay, Samish Bay, and Padilla Bay were also successfully simulated. Both model results and observed data indicated that water surface elevations inside Bellingham Bay are highly correlated to tides. Circulation inside the bay is weak and complex and is affected by various forcing mechanisms, including tides, winds, freshwater inflows, and other local forcing factors. The Bellingham Bay model solution was successfully linked to the NOAA oil spill trajectory simulation model “General NOAA Operational Modeling Environment (GNOME).” Overall, the Bellingham Bay model has been calibrated reasonably well and can be used to provide detailed hydrodynamic information in the bay and adjacent water bodies. While there is room for further improvement with more available data, the calibrated hydrodynamic model provides useful hydrodynamic information in Bellingham Bay and can be used to support sediment transport and water quality modeling as well as assist in the design of nearshore restoration scenarios.

02 PETROLEUM↗

Data Assimilation and Dismantling Waste: A Methodology Customised for Radioactive Inventory Assessment Purposes

Data assimilation is a versatile methodology which is widely used nowadays for estimating complex system variables. The estimation of a given quantity by data assimilation involves taking into account both the observational data and the underlying physical principles governing the system under observation. Let us consider two assumptions, (1) the observational information consists of radioactivity measurements of numerous samples of graphite (or other material) from known points and (2) the physical principles are those of activation under neutron flux during nuclear power plant operation. With these two assumptions, a data-assimilation-like methodology is ready to be applied to radioactive inventory computation for dismantling graphite (or other material). The paper is written for a planned graphite disposal project in France, for the purposes of its inventory forecasting. It introduces specific concepts for activation laws, presents the beginning of the application to Bugey 1 plant pile and proposes an original method for overall error quantification. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Modelling infrasound signal generation from two underground explosions at the Source Physics Experiment using the Rayleigh integral

For this study, we use the Rayleigh integral (RI) as an approximation to the Helmholtz–Kirchoff integral to model infrasound generation and propagation from underground chemical explosions at distances of 250 m out to 5 km as part of the Source Physics Experiment (SPE). Using a sparse network of surface accelerometers installed above ground zero, we are able to accurately create synthetic acoustic waveforms and compare them to the observed data. Although the underground explosive sources were designed to be symmetric, the resulting seismic wave at the surface shows an asymmetric propagation pattern that is stronger to the northeast of the borehole. This asymmetric bias may be attributed to the subsurface geology and faulting of the area and is observed in the acoustic waveforms. We compare observed and modelled results from two of the underground SPE tests with a sensitivity study to evaluate the asymmetry observed in the data. This work shows that it is possible to model infrasound signals from underground explosive sources using the RI and that asymmetries observed in the data can be modelled with this technique.

58 GEOSCIENCES↗

Model Data Archive for Manuscript Titled "Evaluation of a Coupled Surface–Subsurface Hydrologic Model Using Dense Water‑Level Sensors in a Mixed Urban–Rural Watershed"

This archive provides scripts, input files, and datasets used for the implementation and evaluation of a fully coupled surface–subsurface hydrologic model in the Neches River Basin, southeast Texas. The study uses the Advanced Terrestrial Simulator (ATS) to simulate coupled surface–subsurface hydrologic processes over a mixed urban–rural watershed and evaluates model performance using a dense network of 136 in situ water-level sensors, nine U.S. Geological Survey (USGS) stream gauges, and SSEBop-derived evapotranspiration estimates during the period October 2014–June 2024. The workflow is implemented primarily in Python 3 using the Watershed Workflow package. The Jupyter notebooks can be executed using open-source software such as Anaconda JupyterLab or Visual Studio Code. Other data files include TXT, CSV, XML, SHP, TIF, NetCDF, HDF5, and ExodusII files, which can be processed using the provided Python scripts. ATS input files are provided in XML format and can be edited using any commonly used text editor. This archive contains: *Scripts and input files used to generate the ATS model setup, including watershed discretization, mesh generation, parameter mapping, and model configuration. *Jupyter notebooks used for preprocessing observational data, evaluating streamflow, water levels, and evapotranspiration, computing performance metrics, and generating the figures presented in the manuscript. *ATS simulation outputs and processed observational datasets, including OneRain and DD6 water-level sensors, USGS streamflow observations, GIS data, and supporting spatial datasets used throughout the study.

Dense water-level sensor network↗

Quantifying and reducing model-form uncertainties in Reynolds-averaged Navier–Stokes simulations: A data-driven, physics-informed Bayesian approach

Despite their well-known limitations, Reynolds-Averaged Navier–Stokes (RANS) models are still the workhorse tools for turbulent flow simulations in today's engineering analysis, design and optimization. While the predictive capability of RANS models depends on many factors, for many practical flows the turbulence models are by far the largest source of uncertainty. As RANS models are used in the design and safety evaluation of many mission-critical systems such as airplanes and nuclear power plants, quantifying their model-form uncertainties has significant implications in enabling risk-informed decision-making. In this work we develop a data-driven, physics-informed Bayesian framework for quantifying model-form uncertainties in RANS simulations. Uncertainties are introduced directly to the Reynolds stresses and are represented with compact parameterization accounting for empirical prior knowledge and physical constraints (e.g., realizability, smoothness, and symmetry). An iterative ensemble Kalman method is used to assimilate the prior knowledge and observation data in a Bayesian framework, and to propagate them to posterior distributions of velocities and other Quantities of Interest (QoIs). We use two representative cases, the flow over periodic hills and the flow in a square duct, to evaluate the performance of the proposed framework. Both cases are challenging for standard RANS turbulence models. Simulation results suggest that, even with very sparse observations, the obtained posterior mean velocities and other QoIs have significantly better agreement with the benchmark data compared to the baseline results. At most locations the posterior distribution adequately captures the true model error within the developed model form uncertainty bounds. The framework is a major improvement over existing black-box, physics-neutral methods for model-form uncertainty quantification, where prior knowledge and details of the models are not exploited. This approach has potential implications in many fields in which the governing equations are well understood but the model uncertainty comes from unresolved physical processes. - Highlights: • Proposed a physics–informed framework to quantify uncertainty in RANS simulations. • Framework incorporates physical prior knowledge and observation data. • Based on a rigorous Bayesian framework yet fully utilizes physical model. • Applicable for many complex physical systems beyond turbulent flows.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Distance measurements from supernovae and dark energy constraints

Constraints on dark energy from current observational data are sensitive to how distances are measured from Type Ia supernova (SN Ia) data. We find that flux averaging of SNe Ia can be used to test the presence of unknown systematic uncertainties, and yield more robust distance measurements from SNe Ia. We have applied this approach to the nearby+SDSS+ESSENCE+SNLS+HST set of 288 SNe Ia, and the 'Constitution' set of 397 SNe Ia. Combining the SN Ia data with cosmic microwave background anisotropy data from Wilkinson Microwave Anisotropy Probe 5 yr observations, the Sloan Digital Sky Survey baryon acoustic oscillation measurements, the data of 69 gamma-ray bursts (GRBs) , and the Hubble constant measurement from the Hubble Space Telescope project SHOES, we measure the dark energy density function X(z){identical_to}{rho}{sub X}(z)/{rho}{sub X}(0) as a free function of redshift (assumed to be a constant at z>1 or z>1.5). Without the flux averaging of SNe Ia, the combined data using the Constitution set of SNe Ia seem to indicate a deviation from a cosmological constant at {approx}95% confidence level at 0 < or apporx. z < or approx. 0.8; they are consistent with a cosmological constant at {approx}68% confidence level when SNe Ia are flux averaged. The combined data using the nearby+SDSS+ESSENCE+SNLS+HST data set of SNe Ia are consistent with a cosmological constant at 68% confidence level with or without flux averaging of SNe Ia, and give dark energy constraints that are significantly more stringent than that using the Constitution set of SNe Ia. Assuming a flat Universe, dark energy is detected at >98% confidence level for z{<=}0.75 using the combined data with 288 SNe Ia from nearby+SDSS+ESSENCE+SNLS+HST, independent of the assumptions about X(z{>=}1). We quantify dark energy constraints without assuming a flat Universe using the dark energy figure of merit for both X(z) and a dark energy equation-of-state linear in the cosmic scale factor.

99 GENERAL AND MISCELLANEOUS↗

Space‐Time Causal Discovery in Earth System Science: A Local Stencil Learning Approach

Causal discovery tools enable scientists to infer meaningful relationships from observational data, spurring advances in fields as diverse as biology, economics, and climate science. Despite these successes, the application of causal discovery to space-time systems remains immensely challenging due to the high-dimensional nature of the data. For example, in climate sciences, modern observational temperature records over the past few decades regularly measure thousands of locations around the globe. To address these challenges, we introduce Causal Space-Time Stencil Learning (CaStLe), a novel meta-algorithm for discovering causal structures in complex space-time systems. CaStLe leverages regularities in local space-time dependencies to learn governing global dynamics. This local perspective eliminates spurious confounding and drastically reduces sample complexity, making space-time causal discovery practical and effective. For causal discovery, CaStLe flexibly accepts any appropriately adapted time series causal discovery algorithm to recover local causal structures. These advances enable causal discovery of geophysical phenomena that were previously unapproachable, including non-periodic, transient phenomena such as volcanic eruption plumes. Regularities in local space-time dependencies are transformed into informative spatial replicates, which actually improve CaStLe's performance when applied to ever-larger spatial grids. We successfully apply CaStLe to discover the atmospheric dynamics governing the climate response to the 1991 Mount Pinatubo volcanic eruption. We provide validation experiments to demonstrate the effectiveness of CaStLe over existing causal-discovery frameworks on a range of geophysics-inspired benchmarks while identifying the method's limitations and domains where its assumptions may not hold.

Nichol, J. Jake [Univ. of New Mexico, Albuquerque,↗

Stark Widths and Shifts Dependence on the Rest Core Charge of the Emitters within ns-np Transition Arrays

Stark width and shift simultaneous dependence on the upper level ionization potential and rest core charge of the emitter has been evaluated and discussed. It has been verified that the found relations, connecting Stark broadening parameters with upper level ionization potential and rest core charge of the emitters for particular electron temperature and density, can be used for prediction of Stark line width and shift data in case of ions for which observed data, or more detailed calculations, are not yet available. Stark widths and shifts published data are used to demonstrate the existence of other kinds of regularities within similar spectra of different elements and their ionization stages. The emphasis is on the Stark parameter dependence on the upper level ionization potential and on the rest core charge for the lines from similar spectra of multiply charged ions. The found relations connecting Stark widths and shift parameters with upper level ionization potential, rest core charge and electron temperature were used for a prediction of new Stark broadening data, thus avoiding much more complicated procedures.

74 ATOMIC AND MOLECULAR PHYSICS↗

Stark parameter dependence of the rest core charge of the emitters for multiply charged ions spectral lines

Stark width and shift simultaneous dependence on the upper level ionization potential and rest core charge of the emitter has been evaluated and discussed. It has been verified that the found relations, connecting Stark broadening parameters with upper level ionization potential and rest core charge of the emitters for particular electron temperature and density, can be used for prediction of Stark line width and shift data in case of ions for which observed data, or more detailed calculations, are not yet available. Stark widths and shifts published data are used to demonstrate the existence of other kinds of regularities within similar spectra of different elements and their ionization stages. The emphasis is on the Stark parameter dependence on the upper level ionization potential and on the rest core charge for the lines from similar spectra of multiply charged ions. The found relations connecting Stark widths and shift parameters with upper level ionization potential, rest core charge and electron temperature were used for a prediction of new Stark broadening data, thus avoiding much more complicated procedures.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

NEAR-EARTH BINARIES AND TRIPLES: ORIGIN AND EVOLUTION OF SPIN-ORBITAL PROPERTIES

In the near-Earth asteroid population, binary and triple systems have been discovered with mutual orbits that have significant eccentricities as well as large semimajor axes. All known systems with eccentric orbits and all widely separated primary-satellite pairs have rapidly rotating satellites. Here, we study processes that can elucidate the origin of these spin-orbital properties. Binary formation models based on rotational fissioning can reproduce asynchronous satellites on orbits with high eccentricities and a wide range of separations, but do not match observed properties. We explore whether any evolutionary mechanisms can link the spin and orbital parameters expected from post-fission dynamics to those observed today. We investigate four processes: tidal torques, radiative perturbations (BYORP), close planetary encounters, and Kozai oscillations. We find that a combination of post-fission dynamics and tidal evolution can explain nearly all the spin-orbit properties in a sample of nine well-characterized near-Earth binaries and triples. The other mechanisms may act as well but are not required to explain the observed data. Lastly, we describe evolutionary pathways between observed spin-orbital states including synchronous and circular, asynchronous and circular, and asynchronous and eccentric configurations.

79 ASTRONOMY AND ASTROPHYSICS↗

3D ADAPTIVE MESH REFINEMENT SIMULATIONS OF THE GAS CLOUD G2 BORN WITHIN THE DISKS OF YOUNG STARS IN THE GALACTIC CENTER

The dusty, ionized gas cloud G2 is currently passing the massive black hole in the Galactic Center at a distance of roughly 2400 Schwarzschild radii. We explore the possibility of a starting point of the cloud within the disks of young stars. We make use of the large amount of new observations in order to put constraints on G2's origin. Interpreting the observations as a diffuse cloud of gas, we employ three-dimensional hydrodynamical adaptive mesh refinement (AMR) simulations with the PLUTO code and do a detailed comparison with observational data. The simulations presented in this work update our previously obtained results in multiple ways: (1) high resolution three-dimensional hydrodynamical AMR simulations are used, (2) the cloud follows the updated orbit based on the Brackett-γ data, (3) a detailed comparison to the observed high-quality position–velocity (PV) diagrams and the evolution of the total Brackett-γ luminosity is done. We concentrate on two unsolved problems of the diffuse cloud scenario: the unphysical formation epoch only shortly before the first detection and the too steep Brackett-γ light curve obtained in simulations, whereas the observations indicate a constant Brackett-γ luminosity between 2004 and 2013. For a given atmosphere and cloud mass, we find a consistent model that can explain both, the observed Brackett-γ light curve and the PV diagrams of all epochs. Assuming initial pressure equilibrium with the atmosphere, this can be reached for a starting date earlier than roughly 1900, which is close to apo-center and well within the disks of young stars.

79 ASTRONOMY AND ASTROPHYSICS↗

Observational measures of halo properties beyond mass

Different properties of dark matter haloes, including growth rate, concentration, interaction history, and spin, correlate with environment in unique, scale-dependent ways. While these halo properties are not directly observable, galaxies will inherit their host haloes’ correlations with environment. In this paper, we show how these characteristic environmental signatures allow using measurements of galaxy environment to constrain which dark matter halo properties are most tightly connected to observable galaxy properties. Here, we show that different halo properties beyond mass imprint distinct scale-dependent signatures in both the galaxy two-point correlation function and the distribution of distances to galaxies’ kth nearest neighbours, with features strong enough to be accessible even with low-resolution (e.g. grism) spectroscopy at higher redshifts. As an application, we compute observed two-point correlation functions for galaxies binned by half-mass radius at $z$ = 0 from the Sloan Digital Sky Survey, showing that classic galaxy size models (i.e. galaxy size being proportional to halo spin) as well as other recent proposals show significant tensions with observational data. We show that the agreement with observed clustering can be improved with a simple empirical model in which galaxy size correlates with halo growth.

79 ASTRONOMY AND ASTROPHYSICS↗

Tachyon cosmology, supernovae data, and the big brake singularity

We compare the existing observational data on type Ia supernovae with the evolutions of the Universe predicted by a one-parameter family of tachyon models which we have introduced recently [Phys. Rev. D 69, 123512 (2004)]. Among the set of the trajectories of the model which are compatible with the data there is a consistent subset for which the Universe ends up in a new type of soft cosmological singularity dubbed big brake. This opens up yet another scenario for the future history of the Universe besides the one predicted by the standard {lambda}CDM model.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Inferring Correlation Networks from Genomic Survey Data

High-throughput sequencing based techniques, such as 16S rRNA gene profiling, have the potential to elucidate the complex inner workings of natural microbial communities - be they from the world's oceans or the human gut. A key step in exploring such data is the identification of dependencies between members of these communities, which is commonly achieved by correlation analysis. However, it has been known since the days of Karl Pearson that the analysis of the type of data generated by such techniques (referred to as compositional data) can produce unreliable results since the observed data take the form of relative fractions of genes or species, rather than their absolute abundances. Using simulated and real data from the Human Microbiome Project, we show that such compositional effects can be widespread and severe: in some real data sets many of the correlations among taxa can be artifactual, and true correlations may even appear with opposite sign. Additionally, we show that community diversity is the key factor that modulates the acuteness of such compositional effects, and develop a new approach, called SparCC (available at https://bitbucket.org/yonatanf/sparcc), which is capable of estimating correlation values from compositional data. To illustrate a potential application of SparCC, we infer a rich ecological network connecting hundreds of interacting species across 18 sites on the human body. Using the SparCC network as a reference, we estimated that the standard approach yields 3 spurious species-species interactions for each true interaction and misses 60% of the true interactions in the human microbiome data, and, as predicted, most of the erroneous links are found in the samples with the lowest diversity.

59 BASIC BIOLOGICAL SCIENCES↗

High-performance End-to-End Integrity Verification on Big Data Transfer

The scale of scientific data generated by experimental facilities and simulations in high-performance computing facilities has been proliferating with the emergence of IoT-based big data. In many cases, this data must be transmitted rapidly and reliably to remote facilities for storage, analysis, or sharing, for the Internet of Things (IoT) applications. Simultaneously, IoT data can be verified using a checksum after the data has been written to the disk at the destination to ensure its integrity. However, this end-to-end integrity verification inevitably creates overheads (extra disk I/O and more computation). Thus, the overall data transfer time increases. In this article, we evaluate strategies to maximize the overlap between data transfer and checksum computation for astronomical observation data. Specifically, we examine file-level and block-level (with various block sizes) pipelining to overlap data transfer and checksum computation. We analyze these pipelining approaches in the context of GridFTP, a widely used protocol for scientific data transfers. Theoretical analysis and experiments are conducted to evaluate our methods. The results show that block-level pipelining is effective in maximizing the overlap mentioned above, and can improve the overall data transfer time with end-to-end integrity verification by up to 70% compared to the sequential execution of transfer and checksum, and by up to 60% compared to file-level pipelining.

data integrity↗