Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Big data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Using Big Data Technologies for HEP Analysis

The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother. In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by developing a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours, collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis. The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.

Cremonesi, Matteo↗
NASA NTRSDate not supplied

Tracking Critical Thermal Metrics throughout the Life Cycle of a Large Observatory Thermal Model

Observatory thermal models for large, complex missions, such as the Wide Field InfraRed Survey Telescope (WFIRST) mission, produce an immense amount of data to be processed. Configuration management of the model throughout the project life cycle has mainly focused on which versions of the subsystem models form the current observatory level configuration. However, the results produced by the model are not nearly as well tracked and traceable. Given the various states of design maturity for each of the components in the WFIRST design, an updated component model is nearly ready to be integrated at the next higher level of assembly about every month or two. With each subsystem model delivery, the observatory model needs to remove the old component, integrate the new one, execute the model, and inspect the results. Usually, this inspection focuses primarily on the newly integrated component. Recently, a Metric Tracking Spreadsheet was developed to help provide a “big picture” view of the entire observatory highlighting key parameters critical to mission performance. This spreadsheet helps track impacts on subsystems by updates of other subsystems that were not intuitively obvious. Metrics tracked include: absorbed environmental loading (to determine effectiveness of sunshield), temperatures of critical avionics, electrical dissipations, heater power predictions, stability of critical optics, parasitic heat leaks in cryogenic region, high level heat flows between elements, and model run time. Producing this data for the same operational configuration with each model update has helped produce a trail of data to evaluate the impact of model updates. While the metrics selected are specific for WFIRST, other large, complex observatories could be well served to establish their own metrics early in the project life cycle to track to quickly assess the impact of any subsystem on other subsystems or the overall system itself.

Thermal Desktop↗
DOE OSTI2018

Studying language evolution in the age of big data

The increasing availability of large digital corpora of cross-linguistic data is revolutionizing many branches of linguistics. Overall, it has triggered a shift of attention from detailed questions about individual features to more global patterns amenable to rigorous, but statistical, analyses. This engenders an approach based on successive approximations where models with simplified assumptions result in frameworks that can then be systematically refined, always keeping explicit the methodological commitments and the assumed prior knowledge. Therefore, they can resolve disputes between competing frameworks quantitatively by separating the support provided by the data from the underlying assumptions. These methods, though, often appear as a ‘black box’ to traditional practitioners. In fact, the switch to a statistical view complicates comparison of the results from these newer methods with traditional understanding, sometimes leading to misinterpretation and overly broad claims. We describe here this evolving methodological shift, attributed to the advent of big, but often incomplete and poorly curated, data, emphasizing the underlying similarity of the newer quantitative to the traditional comparative methods and discussing when and to what extent the former have advantages over the latter. In this review, we cover briefly both randomization tests for detecting patterns in a largely model-independent fashion and phylolinguistic methods for a more model-based analysis of these patterns. We foresee a fruitful division of labor between the ability to computationally process large volumes of data and the trained linguistic insight declaring worthy prior commitments and interesting hypotheses in need of comparison.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2021

In Situ Inference for Earth System Predictability

Focal Area: Focal Area 3: Insight gleaned from complex simulated data using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2013

Nine-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Final Maps and Results

We present the final nine-year maps and basic results from the Wilkinson Microwave Anisotropy Probe (WMAP) mission. The full nine-year analysis of the time-ordered data provides updated characterizations and calibrations of the experiment. We also provide new nine-year full sky temperature maps that were processed to reduce the asymmetry of the effective beams. Temperature and polarization sky maps are examined to separate cosmic microwave background (CMB) anisotropy from foreground emission, and both types of signals are analyzed in detail.We provide new point source catalogs as well as new diffuse and point source foreground masks. An updated template-removal process is used for cosmological analysis; new foreground fits are performed, and new foreground reduced are presented.We nowimplement an optimal C(exp -1)1 weighting to compute the temperature angular power spectrum. The WMAP mission has resulted in a highly constrained Lambda-CDM cosmological model with precise and accurate parameters in agreement with a host of other cosmological measurements. When WMAP data are combined with finer scale CMB, baryon acoustic oscillation, and Hubble constant measurements, we find that big bang nucleosynthesis is well supported and there is no compelling evidence for a non-standard number of neutrino species (N(sub eff) = 3.84 +/- 0.40). The model fit also implies that the age of the universe is (sub 0) = 13.772 +/- 0.059 Gyr, and the fit Hubble constant is H(sub 0) = 69.32 +/- 0.80 km/s/ Mpc. Inflation is also supported: the fluctuations are adiabatic, with Gaussian random phases; the detection of a deviation of the scalar spectral index from unity, reported earlier by the WMAP team, now has high statistical significance (n(sub s) = 0.9608+/-0.0080); and the universe is close to flat/Euclidean (Omega = −0.0027+0.0039/−0.0038). Overall, the WMAP mission has resulted in a reduction of the cosmological parameter volume by a factor of 68,000 for the standard six-parameter Lambda-CDM model, based on CMB data alone. For a model including tensors, the allowed seven-parameter volume has been reduced by a factor 117,000. Other cosmological observations are in accord with the CMB predictions, and the combined data reduces the cosmological parameter volume even further.With no significant anomalies and an adequate goodness of fit, the inflationary flat Lambda-CDM model and its precise and accurate parameters rooted in WMAP data stands as the standard model of cosmology.

dark matter↗
DOE OSTI2022

GeoThermalCloud: Cloud Fusion of Big Data and Multi-Physics Models using Machine Learning for Discovery, Exploration and Development of Hidden Geothermal Resources

Geothermal exploration and production are challenging, expensive and risky. The GeoThermalCloud uses Machine Learning to predict the location of hidden geothermal resources. This submission includes a training dataset for the GeoThermalCloud neural network. Machine Learning for Discovery, Exploration, and Development of Hidden Geothermal Resources.

15 GEOTHERMAL ENERGY↗
DOE OSTI2020

Nucleon-Nucleon Scattering Up to N5LO in Chiral Effective Field Theory

During the past few decades a large effort has been made toward describing the NN interaction in the framework of chiral Effective Field Theory (EFT). The main idea is to exploit the symmetries of QCD to obtain an effective theory for low energy nuclear systems. In 2003, the first accurate charge-dependent NN potential in this scheme was developed and it has been applied to many ab-initio calculations, opening the possibility to study nuclear systems in a systematic and accurate way. It was shown that the fourth order (N3LO) was necessary and sufficient to describe the NN scattering data with a χ 2 /d.o.f on the order of so-called high precision potentials. However the systematics of chiral EFT also allow to relate two- and many-body interactions in a well-defined way. Since many-body forces make their first appearance at higher order, they are substantially smaller than their two-body counterparts, but may never-the-less be crucial for some processes. Thus, there are observables where they can have a big impact and, for example, there are indications that they solve the long standing Ay puzzle of N-d scattering. The last few years, have also seen substantial progress toward higher orders of chiral EFT which was motivated by the fact that only three-body forces of rather high order may solve some outstanding issues in microscopic nuclear structure and reactions. In this chapter we will review the latest contributions of the authors to development of chiral EFT based potentials up to N4LO as well as first calculations conducted for NN scattering at N5LO.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Indirect determination of the astrophysical S factor for the Li 6 ( p , γ ) Be 7 reaction using the asymptotic normalization coefficient method

The 6 Li(p,γ) 7 Be cross section influences a variety of astrophysical scenarios, including big-bang and stellar nucleosynthesis. In recent years, conflicting results of direct measurements have been published, reporting contradictory low-energy trends. To shed light on the contradiction between the existing data sets, the reaction was studied using the asymptotic normalization coefficient (ANC) technique which was up-to-now never used for this reaction. To derive the ANC, the 6 Li( 3 He,d) 7 Be transfer reaction, studied at the Department of Physics and Astronomy of the University of Catania and at the John. D. Fox Superconducting Accelerator Laboratory at Florida State University, was re-analyzed, focusing on the proton transfer mechanism [the α transfer process is discussed by Kiss et al. [Phys. Lett. B 807, 135606 (2020)]. Here, the energy of the 3 He beam impinging on a 6 Li target was E lab = 3 MeV and E lab = 5 MeV. The yield of the emitted deuterons was measured with high precision by using silicon ΔE – E telescopes.

6 ≤ A ≤ 19↗
DOE OSTI2013

NINE-YEAR WILKINSON MICROWAVE ANISOTROPY PROBE (WMAP) OBSERVATIONS: FINAL MAPS AND RESULTS

We present the final nine-year maps and basic results from the Wilkinson Microwave Anisotropy Probe (WMAP) mission. The full nine-year analysis of the time-ordered data provides updated characterizations and calibrations of the experiment. We also provide new nine-year full sky temperature maps that were processed to reduce the asymmetry of the effective beams. Temperature and polarization sky maps are examined to separate cosmic microwave background (CMB) anisotropy from foreground emission, and both types of signals are analyzed in detail. We provide new point source catalogs as well as new diffuse and point source foreground masks. An updated template-removal process is used for cosmological analysis; new foreground fits are performed, and new foreground-reduced CMB maps are presented. We now implement an optimal C {sup –1} weighting to compute the temperature angular power spectrum. The WMAP mission has resulted in a highly constrained ΛCDM cosmological model with precise and accurate parameters in agreement with a host of other cosmological measurements. When WMAP data are combined with finer scale CMB, baryon acoustic oscillation, and Hubble constant measurements, we find that big bang nucleosynthesis is well supported and there is no compelling evidence for a non-standard number of neutrino species (N {sub eff} = 3.84 ± 0.40). The model fit also implies that the age of the universe is t {sub 0} = 13.772 ± 0.059 Gyr, and the fit Hubble constant is H {sub 0} = 69.32 ± 0.80 km s{sup –1} Mpc{sup –1}. Inflation is also supported: the fluctuations are adiabatic, with Gaussian random phases; the detection of a deviation of the scalar spectral index from unity, reported earlier by the WMAP team, now has high statistical significance (n{sub s} = 0.9608 ± 0.0080); and the universe is close to flat/Euclidean (Ω{sub k} = -0.0027{sup +0.0039}{sub -0.0038}). Overall, the WMAP mission has resulted in a reduction of the cosmological parameter volume by a factor of 68,000 for the standard six-parameter ΛCDM model, based on CMB data alone. For a model including tensors, the allowed seven-parameter volume has been reduced by a factor 117,000. Other cosmological observations are in accord with the CMB predictions, and the combined data reduces the cosmological parameter volume even further. With no significant anomalies and an adequate goodness of fit, the inflationary flat ΛCDM model and its precise and accurate parameters rooted in WMAP data stands as the standard model of cosmology.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2020

Scanning Probe Microscopy in the Information Age

Scanning probe microscopy (SPM) has brought scientific exploration, visualization, and control to nanoscale. A plethora of measurement approaches and techniques devised around SPM enabled probing structure and functionalities crucial for virtually all areas of science. While high-resolution electron microscopy techniques preceded scanning probes, SPM effectively opened the door to probing the structure–property relationship on length scales spanning 100s of micrometers down to a single atom. As the complexity of the material systems and the scientific questions increased, so too did the SPM techniques. Substantial efforts have been dedicated toward the development of better instrumentation platforms, probes, and novel spectroscopic measurements; however, many opportunities related to data processing and acquisition in SPM are just being realized now. This chapter will describe the recent progress in data-driven SPM by giving a broad overview of the field, the foundation of the scanning tunneling microscope (STM) and the atomic force microscope (AFM), as well as a description behind the operating principles and dynamic modes. A comprehensive overview of the multi-frequency techniques, Band excitation, and general acquisition mode SPM is given, setting the context for the application of big data, machine learning, and artificial intelligence in the field of dynamic SPM.

Collins, Liam↗
DOE OSTI2018

Machine Learning and Social Media to Mine and Disseminate Big Scientific Data

One of the challenges in supplying the communities with wider access to scientific databases is the need for knowledge of database languages like Structured Query Language (SQL). Although the SQL language has been published in many forms, not everybody is able to write SQL queries. Another challenge is that it might not be practical to make the public aware of the structure of databases. There is a need for novice users to query relational databases using their natural language. To solve this problem, many natural language interfaces to structured databases have been developed. The goal is to provide a more intuitive method for generating database queries and delivering responses. Through social media, which makes it possible to interact with a wide section of the population, and with the help of natural language processing, researchers at the Atmospheric Radiation Measurement (ARM) Data Center at Oak Ridge National Laboratory (ORNL) have developed a concept to enable easy search and retrieval of data from several environmental data centers for the scientific community through social media.Using a machine learning framework that maps natural language text to thousands of datasets, instruments, variables, and data streams, the prototype system would allow users to request data through Twitter and receive a link (via tweet) to applicable data results on the project's search catalog tailored to their key words. This automated identification of relevant data from various petascale archives at ORNL could increase convenience, access, and use of the project's data by the broader community. In this paper we discuss how some data-intensive projects at ORNL are using innovative ways to help in data discovery.

Devarakonda, Ranjeet↗
DOE OSTI2014

Traffic information computing platform for big data

Big data environment create data conditions for improving the quality of traffic information service. The target of this article is to construct a traffic information computing platform for big data environment. Through in-depth analysis the connotation and technology characteristics of big data and traffic information service, a distributed traffic atomic information computing platform architecture is proposed. Under the big data environment, this type of traffic atomic information computing architecture helps to guarantee the traffic safety and efficient operation, more intelligent and personalized traffic information service can be used for the traffic information users.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Major Revisions in Arthropod Phylogeny Through Improved Supermatrix, With Support for Two Possible Waves of Land Invasion by Chelicerates

Deep phylogeny involving arthropod lineages is difficult to recover because the erosion of phylogenetic signals over time leads to unreliable multiple sequence alignment (MSA) and subsequent phylogenetic reconstruction. One way to alleviate the problem is to assemble a large number of gene sequences to compensate for the weakness in each individual gene. Such an approach has led to many robustly supported but contradictory phylogenies. A close examination shows that the supermatrix approach often suffers from two shortcomings. The first is that MSA is rarely checked for reliability and, as will be illustrated, can be poor. The second is that, to alleviate the problem of homoplasy at the third codon position of protein-coding genes due to convergent evolution of nucleotide frequencies, phylogeneticists may remove or degenerate the third codon position but may do it improperly and introduce new biases. We performed extensive reanalysis of one of such “big data” sets to highlight these two problems, and demonstrated the power and benefits of correcting or alleviating these problems. Our results support a new group with Xiphosura and Arachnopulmonata (Tetrapulmonata + Scorpiones) as sister taxa. This favors a new hypothesis in which the ancestor of Xiphosura and the extinct Eurypterida (sea scorpions, of which many later forms lived in brackish or freshwater) returned to the sea after the initial chelicerate invasion of land. Our phylogeny is supported even with the original data but processed with a new “principled” codon degeneration. We also show that removing the 1673 codon sites with both AGN and UCN codons (encoding serine) in our alignment can partially reconcile discrepancies between nucleotide-based and AA-based tree, partly because two sequences, one with AGN and the other with UCN, would be identical at the amino acid level but quite different at the nucleotide level.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2018

An Interactive Visualization Tool for Large-Scale Building Stock Modeling

Recent advancements in data science and high-performance computing are making it easier to run millions of building simulations, but meaningful visualization of such large datasets remains a challenge. This paper presents a new tool developed to view the results of large-scale OpenStudio simulations of national, regional, or local building stocks. The tool processes millions of simulations to calculate measure savings, utility bills, carbon emissions, primary energy, and cost-effectiveness metrics at a high geographic resolution. Interactive visualizations of the building characteristics, consumption, and measure savings data include proportional symbol maps and histogram plots and can be filtered by any building characteristic.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2018

End-System Aware Peak Link Utilization Transport for Big Data Transfer over Dedicated Connections

High-performance networks provisioning 10/40-Gbps dedicated channels have been rapidly developed andincreasingly deployed to support big data transfers on the order of terabytes or petabytes in large-scale network-intensiveapplications. However, end users have not seen corresponding increase in application throughput mainly because traditional end-to-end transport methods are not optimized for such connections. New congestion or flow control mechanisms are desirable to meet the challenges brought to transport protocol design by dedicated connections. The advent and proliferation of multi-core processors make it now possible to improve application throughput by providing multiple processing and network resources to a single data transfer. Based on the existing PLUT method, we proposed a new dynamic buffer management scheme and an automatic parallelism tuning mechanism to achieve the maximal bottleneck rate. The performance superiority of PLUT is justified by the experimental results collected over a local 10Gbps dedicated connection.

Rao, Nageswara↗
DOE OSTI2021

Big PanDa Workflow Management on Titan for High Energy and Nuclear Physics and for Future Extreme Scale Scientific Application

Over a three year period, from 2016-2019, this project demonstrated the scientific benefits of integrating the Titan supercomputer at Oak Ridge Leadership Computing Facility into traditional high throughput grid based distributed computing systems managed by PanDA, the workflow management system used for the execution of all distributed computing applications by the ATLAS experiment at the Large Hadron Collider. PanDA manages millions of batch jobs daily at hundreds of clusters worldwide on request by thousands of physicist users, and processes more than an exabyte of data annually using grid middleware. High levels of operational use of Titan was sustained by PanDA in order to meet the physics goals of ATLAS. The success of this project led to the use of other supercomputers worldwide by ATLAS, and to the adoption of PanDA by other experiments and other scientists. Multiple innovative operational and computer science research goals were achieved supporting the use of supercomputers for scientific domains with large scale distributed data and distributed processing needs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Metagenomics, the study of all microbial species cohabitants in an environment, usually produces large amount of sequence data varying from several GBs to a few TBs. Analyzing metagenomics data includes both data-intensive and compute-intensive steps, making the entire process hard to scale. Here we aim to optimize a metagenomics application that partitions the shortgun metagenomics sequences based on their species of origin. Our solution combines MapReduce-based BioPig analytic toolkit with MPI to provide scalability in respective to both data and compute. We also made some improvements to the existing BioPig toolkit by using simplified data types and compressed k-mer storage. These optimizations leads up to 193× speedup for the computing-intensive step and 9.6× speedup over the entire pipeline. Our optimized application is also capable of processing datasets that are 16 times larger on the same hardware platform. These conclusions indicate integrating heterogeneous technologies such as Hadoop and MPI is quite efficient to solve large genomics problems that are both data-intensive and compute-intensive.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2014

New determination of the {sup 2}H(d,p){sup 3}H and {sup 2}H(d,n){sup 3}He reaction rates at astrophysical energies

The cross sections of the {sup 2}H(d,p){sup 3}H and {sup 2}H(d,n){sup 3}He reactions have been measured via the Trojan Horse method applied to the quasi-free {sup 2}H({sup 3}He,p {sup 3}H){sup 1}H and {sup 2}H({sup 3}He,n {sup 3}He){sup 1}H processes at 18 MeV off the proton in {sup 3}He. For the first time, the bare nucleus S(E) factors have been determined from 1.5 MeV, across the relevant region for standard Big Bang nucleosynthesis, down to the thermal energies of deuterium burning in the pre-main-sequence (PMS) phase of stellar evolution, as well as of future fusion reactors. Both the energy dependence and the absolute value of the S(E) factors deviate by more than 15% from the available direct data and existing fitting curves, with substantial variations in the electron screening by more than 50%. As a consequence, the reaction rates for astrophysics experience relevant changes, with a maximum increase of up to 20% at the temperatures of the PMS phase. From a recent primordial abundance sensitivity study, it turns out that the {sup 2}H(d,n){sup 3}He reaction is quite influential on {sup 7}Li, and the present change in the reaction rate leads to a decrease in its abundance by up to 10%. The present reaction rates have also been included in an updated version of the FRANEC evolutionary code to analyze their influence on the central deuterium abundance in PMS stars with different masses. The largest variation of about 10%-15% pertains to young stars (≤1 Myr) with masses ≥1 M {sub ☉}.

79 ASTRONOMY AND ASTROPHYSICS↗