Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data services”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 1 Sensor Data v1-2

This is the version 1-2 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments.L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds and out-of-service flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project**This dataset includes:- An overall dataset README file that describes the current version, gives citation and contact information, etc.- Site- and year-specific folders, each holding up to 12 CSV (comma separated value) data files for each site and plot in that year.- Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site.- Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes.Please see v1-2 TEMPEST L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning.The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods.* Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021* TEMPEST 1: June 22, 2022* TEMPEST 2: June 6-7, 2023* TEMPEST 3: June 11-13, 2024

54 ENVIRONMENTAL SCIENCES↗

NASA GISS Surface Temperature (GISTEMP) Analysis

The NASA GISS Surface Temperature (GISTEMP) analysis provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. The input data that the GISTEMP Team use for the analysis, collected by many national meteorological services around the world, are the adjusted data of the Global Historical Climatology Network (GHCN) Vs. 3 (this represents a change from prior use of unadjusted Vs. 2 data) (Peterson and Vose, 1997 and 1998), United States Historical Climatology Network (USHCN) data, and SCAR (Scientific Committee on Antarctic Research) data from Antarctic stations. Documentation of the basic analysis method is provided by Hansen et al. (1999), with several modifications described by Hansen et al. (2001). The GISS analysis is updated monthly, however CDIAC's presentation of the data here is updated annually.For access to the data files, click this link to the CDIAC data transition website: http://cdiac.ess-dive.lbl.gov/trends/temp/hansen/hansen.html

(GISTEMP)↗

Surface Studies of HSLA Steel after Electrochemical Corrosion in Supercritical CO2-H2O Environment

In aqueous phase saturated with CO2, X-65 sample underwent general corrosion with formation of FeCO3. In supercritical CO2 containing water phase, two major regions are present on the sample surface after the EIS experiment. One region covered with corrosion products identified as FeCO3 and the other containing Fe, oxygen, and carbon-rich islands embedded in metal matrix identified as {alpha}-Fe. Precipitation of FeCO3 from Fe2+ and CO3 2- is responsible for formation of passive layer in oxygen-deficient, CO2 rich aqueous environment. Mechanisms of corrosion degradation occurring in supercritical CO2 as a function. Transport of supercritical CO{sub 2} is a critical element for carbon capture from fossil fuel power plants and underground sequestration. Although acceptable levels of water in supercritical CO{sub 2} (up to {approx} 5 x 10{sup -4}g/dm{sup 3}) have been established, their effects on the corrosion resistance of pipeline steels are not fully known. Moreover, the presence of SO{sub 2}, O{sub 2} impurities in addition to the water can make the fluid more corrosive and, therefore, more detrimental to service materials. Also, in this case, limited data are available on materials performance of carbon steels. to advance this knowledge, other service alloys are being investigated in the high pressure high temperature cell containing impure CO{sub 2} fluids using reliable non-destructive in-situ electrochemical methods. The electrochemical results are being augmented by a number of surface analyses of the corroded surfaces.

20 FOSSIL-FUELED POWER PLANTS↗

Inferring Netflow Data

Netflow data is a specific format for looking at network traffic. With this format, many security tests are easy to run. Cloud Service Providers produce logs that do not contain all of the content that Netflow data contains. There is a possibility that the fields that are in Netflow data but absent in Cloud Service Provider logs can be inferred from the fields that are present in the Cloud Service Provider logs. By inferring the missing data fields, the same test that can be run with Netflow data would be possible to run on the logs produced by Cloud Service Providers. Cloud Service Providers include Amazon Web Services and Box.com. There are multiple different types of Cloud Services, and each provider handles them differently and produces different logs. There are IaaS (meaning infrastructure as a service), Saas (meaning software as a service), and Paas (meaning platform as a service). Each provides a different use for the user.

97 MATHEMATICS AND COMPUTING↗

Mesh-free data transfer algorithms for partitioned multiphysics problems: Conservation, accuracy, and parallelism

In this study we analyze and extend mesh-free algorithms for three-dimensional data transfer problems in partitioned multiphysics simulations. We first provide a direct comparison between a mesh-based weighted residual method using the common-refinement scheme and two mesh-free algorithms leveraging compactly supported radial basis functions: one using a spline interpolation and one using a moving least square reconstruction. Through the comparison we assess both the conservation and accuracy of the data transfer obtained from each of the methods. We do so for a varying set of geometries with and without curvature and sharp features and for functions with and without smoothness and with varying gradients. Our results show that the mesh-based and mesh-free algorithms are complementary with cases where each was demonstrated to perform better than the other. We then focus on the mesh-free methods by developing a set of algorithms to parallelize them based on sparse linear algebra techniques. This includes a discussion of fast parallel radius searching in point clouds and restructuring the interpolation algorithms to leverage data structures and linear algebra services designed for large distributed computing environments. The scalability of our new algorithms is demonstrated on a leadership class computing facility using a set of basic scaling studies. Finally, these scaling studies show that for problems with reasonable load balance, our new algorithms for both spline interpolation and moving least square reconstruction demonstrate both strong and weak scalability using more than 100,000 MPI processes with billions of degrees of freedom in the data transfer operation.

97 MATHEMATICS AND COMPUTING↗

Managing high-bandwidth real-time data storage

There exist certain systems which generate real-time data at high bandwidth, but do not necessarily require the long-term retention of that data in normal conditions. In some cases, the data may not actually be useful, and in others, there may be too much data to permanently retain in long-term storage whether it is useful or not. However, certain portions of the data may be identified as being vitally important from time to time, and must therefore be retained for further analysis or permanent storage without interrupting the ongoing collection of new data. We have developed a system, Mahanaxar, intended to address this problem. It provides quality of service guarantees for incoming real-time data streams and simultaneous access to already-recorded data on a best-effort basis utilizing any spare bandwidth. It has built in mechanisms for reliability and indexing, can scale upwards to meet increasing bandwidth requirements, and handles both small and large data elements equally well. We will show that a prototype version of this system provides better performance than a flat file (traditional filesystem) based version, particularly with regard to quality of service guarantees and hard real-time requirements.

99 GENERAL AND MISCELLANEOUS↗

PV O&M Cost Model [SWR-21-14]

This tool presents a method for calculating costs associated with the operation and maintenance (O&M) of photovoltaic (PV) systems. The tool compiles details regarding the cost and frequency of multiple O&M services to estimate annual O&M costs ($/year) for each year of an analysis period, the net present value ($) of life cycle costs accumulated over the analysis period, and the reserve account amount ($) that might be required to fund unexpected repairs. The reserve account includes parts inventory and is important for providing a source of funds to make repairs quickly and avoid lost production. This method allows a detailed selection of services to perform based on system size, market served (e.g., residential, commercial, or utility), type and configuration of system components (e.g., micro-, string, or central inverter), and site and environmental conditions (e.g., pollen, bird populations) which is an improvement over simple per unit valuations of O&M costs ($/kW/year). This model also distinguishes costs that vary from year to year and increase at different rates over time as modeled by heuristic failure distributions (e.g., Weibull or Lognormal distribution) based on actuarial data for many of the services.

Walker, Andy↗

CMS distributed data analysis with CRAB3

The CMS Remote Analysis Builder (CRAB) is a distributed workflow management tool which facilitates analysis tasks by isolating users from the technical details of the Grid infrastructure. Throughout LHC Run 1, CRAB has been successfully employed by an average of 350 distinct users each week executing about 200,000 jobs per day.CRAB has been significantly upgraded in order to face the new challenges posed by LHC Run 2. Components of the new system include 1) a lightweight client, 2) a central primary server which communicates with the clients through a REST interface, 3) secondary servers which manage user analysis tasks and submit jobs to the CMS resource provisioning system, and 4) a central service to asynchronously move user data from temporary storage in the execution site to the desired storage location. Furthermore, the new system improves the robustness, scalability and sustainability of the service.Here we provide an overview of the new system, operation, and user support, report on its current status, and identify lessons learned from the commissioning phase and production roll-out.

97 MATHEMATICS AND COMPUTING↗

The metagenomics RAST server - a public resource for the automatic phylogenetic and functional analysis of metagenomes.

Random community genomes (metagenomes) are now commonly used to study microbes in different environments. Over the past few years, the major challenge associated with metagenomics shifted from generating to analyzing sequences. High-throughput, low-cost next-generation sequencing has provided access to metagenomics to a wide range of researchers. A high-throughput pipeline has been constructed to provide high-performance computing to all researchers interested in using metagenomics. The pipeline produces automated functional assignments of sequences in the metagenome by comparing both protein and nucleotide databases. phylogenetic and functional summaries of the metagenomes are generated, and tools for comparative metagenomics are incorporated into the standard views. user access is controlled to ensure data privacy, but the collaborative environment underpinning the service provides a framework for sharing databasets between multiple users. In the metagenomics RAST, all users retain full control of their data, and everything is available for download in a variety of formats. The open-source metagenomics RAST service provides a new paradigm for the annotation and analysis of metagenomes. With built-in support for multiple data sources and a back end that houses abstract data types, the metagenomics RAST is stable, extensible, and freely available to all researchers. This service has removed one of the primary bottlenecks in metagenome sequence analysis--the available of high-performance computing for annotating the data.

59 BASIC BIOLOGICAL SCIENCES↗

The Role of National Research and Education Networks Providing Connectivity and Advanced Network Services to Virtual Communities in Collaborative R&E Projects. CUDI: The Mexican Case Part 1, , In: ISUM 2015: High Performance Computer Applications

This paper describes the role of National Research and Education Networks (NRENs) and their aim to provide a seamless, large scale, high-speed, low-latency network to accelerate scientific and education discovery for national, regional or global collaborative projects that are dependent on the use of modern high performance distributed computing systems (HPDCS) to process mathematical models, produce new knowledge and drive innovation when resolving complex problems. NRENs have become the mainstream of communication and collaboration to integrate the efforts of large geographically distributed teams of Virtual Communities (VC) that - train, teach and learn-, -create, use and share- knowledge, data, information and communications technology resources (e.g. computer and information systems, middleware, tools, instruments, data sets, repositories, applications and services) building e-infrastructures necessary to meet the collaborative projects’ goals. The paper focuses on the current status of the Mexican NREN: CUDI, its future organization and political support, necessary to provide seamless, large scale, high-speed, low latency and secure national backbone and international links providing connectivity, advanced network services and technical support to VCs and their e-infrastructures in collaborative R&E projects within the country, the LAC region and worldwide.

Computer Science↗

Next Generation Integrated Environment for Collaborative Work Across Internets

We are now well-advanced in our development, prototyping and deployment of a high performance next generation Integrated Environment for Collaborative Work. The system, aimed at using the capability of ESnet and Internet2 for rapid data exchange, is based on the Virtual Room Videoconferencing System (VRVS) developed by Caltech. The VRVS system has been chosen by the Internet2 Digital Video (I2-DV) Initiative as a preferred foundation for the development of advanced video, audio and multimedia collaborative applications by the Internet2 community. Today, the system supports high-end, broadcast-quality interactivity, while enabling a wide variety of clients (Mbone, H.323) to participate in the same conference by running different standard protocols in different contexts with different bandwidth connection limitations, has a fully Web-integrated user interface, developers and administrative APIs, a widely scalable video network topology based on both multicast domains and unicast tunnels, and demonstrated multiplatform support. This has led to its rapidly expanding production use for national and international scientific collaborations in more than 60 countries. We are also in the process of creating a 'testbed video network' and developing the necessary middleware to support a set of new and essential requirements for rapid data exchange, and a high level of interactivity in large-scale scientific collaborations. These include a set of tunable, scalable differentiated network services adapted to each of the data streams associated with a large number of collaborative sessions, policy-based and network state-based resource scheduling, authentication, and optional encryption to maintain confidentiality of inter-personal communications. High performance testbed video networks will be established in ESnet and Internet2 to test and tune the implementation, using a few target application-sets.

97 MATHEMATICS AND COMPUTING↗

Real-Time Discovery Services over Large, Heterogeneous and Complex Healthcare Datasets Using Schema-Less, Column-Oriented Methods

We present a service platform for schema-leess exploration of data and discovery of patient-related statistics from healthcare data sets. The architecture of this platform is motivated by the need for fast, schema-less, and flexible approaches to SQL-based exploration and discovery of information embedded in the common, heterogeneously structured healthcare data sets and supporting components (electronic health records, practice management systems, etc.) The motivating use cases described in the paper are clinical trials candidate discovery, and a treatment effectiveness analysis. Following the use cases, we discuss the key features and software architecture of the platform, the underlying core components (Apache Parquet, Drill, the web services server), and the runtime profiles and performance characteristics of the platform. We conclude by showing dramatic speedup with some approaches, and the performance tradeoffs and limitations of others.

Apache Drill Apache Parquet Column Oriented Stores↗

A Climatology and Life‐Cycle Characteristics of Atmospheric Fronts and Their Associated Precipitation

Abstract Atmospheric fronts are one of the main sources of mid‐latitude variability. We employ a novel method for identifying and tracking fronts and frontal precipitation. Thermal and dynamical variables are used to identify fronts as areal objects in space, which are tracked in time using the open‐source TempestExtremes software package. Precipitation objects are co‐located to identify frontal precipitation. The method is subjected to validation and sensitivity tests using manually curated data from the National Weather Service. Climatologies of fronts and frontal precipitation are computed from reanalysis and observations; fronts are present upwards of 14% of the time in the storm tracks, and represent the majority (up to 90%) of total and extreme precipitation. Novel aspects of the method are showcased through the lifetime characteristics of fronts across North America. Three sets of warm and cold fronts were discovered, and their duration, distance‐traveled, and translation velocity are examined. Plain Language Summary Mid‐latitude low‐pressure systems and weather fronts are important for our day‐to‐day experience of weather events, particularly in the mid‐latitudes. This work makes use of standardized atmospheric data and creates a method of automatically tracking these important atmospheric features and their precipitation to quantify their relative role in global precipitation. Weather fronts are persistent in the mid‐latitudes and are associated with the majority of precipitation–particularly the most intense precipitation. Trajectories of fronts over North America are categorized to create a set of archetypal fronts that occur in that region. The differences between these types of fronts are characterized. Key Points An automated, efficient, and skillful frontal detection algorithm is developed and validated Fronts contribute a larger fraction of extreme precipitation than all precipitation in mid‐latitude storm tracks Fronts across North America have substantial variation in characteristics depending on their origin location

extratropical cyclone↗

Resilient Information Architecture Platform for Smart Grid (RIAPS)

A number of emerging trends will substantially alter the operation and control of the electric grid over the next several decades. These trends include ensuring resiliency under severe weather events, increasing integration of renewable electricity generation, supporting changing electricity demand patterns, and the improving cost effectiveness of distributed energy resources. To address these challenges, the future “Smart Grid” management will need to transition from centralized to coordinated distributed control paradigm. Reliable operation of the Smart Grid depends on distributed intelligence realized through software applications that run on distributed computing devices attached to the power system to collect data and collaboratively manage resources. However, much of the existing software for Smart Grid-enabled devices is either proprietary or developed with custom solutions, which limits interoperability among the heterogeneous devices and hinders the ability to manage system-level reliability, security, and resiliency requirements. Additionally, this approach makes Smart Grid applications hard to maintain, evolve, verify, and replace; resulting in high development and deployment costs. Further development of the Smart Grid requires a reusable software base-layer to move from hard-coded functionality to a plug-and-play architecture capable of managing system-level objectives and constraints in addition to providing consistent common services across heterogeneous devices and applications. Vanderbilt University, in collaboration with North Carolina State University and Washington State University has developed a foundation ‘software platform’ for developing and deploying robust, reliable, effective and secure software applications for the Smart Grid. The Resilient Information Architecture Platform for the Smart Grid (RIAPS) provides core services for building effective and powerful smart grid applications. It offers unique services for real-time data dissemination, fault tolerance, and coordination across apps distributed over the network.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Advance Resource Provisioning in Bulk Data Scheduling

Today?s scientific and business applications generate mas- sive data sets that need to be transferred to remote sites for sharing, processing, and long term storage. Because of increasing data volumes and enhancement in current net- work technology that provide on-demand high-speed data access between collaborating institutions, data handling and scheduling problems have reached a new scale. In this paper, we present a new data scheduling model with ad- vance resource provisioning, in which data movement operations are defined with earliest start and latest comple- tion times. We analyze time-dependent resource assign- ment problem, and propose a new methodology to improve the current systems by allowing researchers and higher-level meta-schedulers to use data-placement as-a-service, so they can plan ahead and submit transfer requests in advance. In general, scheduling with time and resource conflicts is NP-hard. We introduce an efficient algorithm to organize multiple requests on the fly, while satisfying users? time and resource constraints. We successfully tested our algorithm in a simple benchmark simulator that we have developed, and demonstrated its performance with initial test results.

97 MATHEMATICS AND COMPUTING↗

Materials data analytics for 9% Cr family steel

A materials data analytics (MDA) methodology was developed in this study to evaluate publicly available information on 9% Cr family steel and to handle nonlinear relationships and the sparsity in materials data for this alloy class. The overarching goal is to accelerate the design process as well as to reduce the time and expense associated with qualification testing of new alloys for fossil energy applications. Data entries in the analyzed data set for 82 iron–base alloy compositions, several processing parameters, and results of tensile mechanical tests selected for this study were arranged in 34 columns by 915 rows. While detailed microstructural information was not available, it is assumed that the compositional space for the 9 to 12% Cr steels is limited such that all data entries have a tempered martensitic microstructure during service. Establishing a hierarchy of first–order trends in the publicly available data requires the MDA to filter out the biases. Complexity of the phase transformations and microstructure evolution in the multicomponent alloys (using 21 chemical elements) with major influence on mechanical properties, leads to inefficiency in direct application of unbiased linear regression across the entire data space. To address the nonlinearity, analyses of tensile data were performed in composition–based clusters. Clusters corresponding to moderately frequent patterns and maximized information gain were further refined by using p–norm distance measures, matching the alloy classification groups adopted by industry. Lastly, the evolutionary method of propagating an ensemble of competing cluster–based models proved to be a viable option in dealing with scarce, multidimensional data.

36 MATERIALS SCIENCE↗

Lake-Effect Snowstorm Events and Associated Snowfall Totals Integrated from NOAA Storm Reports, ERA5, and HRRR for the Laurentian Great Lakes (1997–2024)

Lake-effect snowstorms are localized, impactful winter weather phenomena that can generate substantial snowfall totals and pose significant challenges for forecasting, transportation, and regional infrastructure. To support the analysis and modeling of these events, this dataset compiles observational reports of lake-effect snowstorms alongside corresponding snowfall estimates derived from gridded atmospheric datasets. The observational component of the data originates from the National Weather Service (NWS) winter storm report, subset to lake-effect snow event type, covering 1997–2024. For each lake-effect snow event, this data provides the impacted county, event start and end datetimes at an hourly resolution, as well as relevant storm narratives. The complementary reanalysis-derived data is sourced from European Centre for Medium-Range Weather Forecasts (ECMWF) Reanalysis 5 (ERA5) and High-Resolution Rapid Refresh (HRRR) gridded data. For both gridded datasets, the maximum total snowfall (in units mm) was extracted, constrained by the county and datetimes specified by the observational report. ERA5 data covers the entire observational period (1997–2024), whereas HRRR data is only available from November 2016 – December 2024. Three CSV files are provided here: (1) the observational lake-effect snow event report, (2) ERA5 maximum snowfall detections for each event, and (3) HRRR maximum snowfall detections for each event. Relevant data from the observational files, such as impacted state and county, event datetimes, and event IDs, were included for convenience. Users can inspect and visualize the data using tools such as Microsoft Excel and Python pandas/matplotlib packages. This dataset may support a variety of applications, including climatological analyses of lake-effect snowfall, evaluation of snowfall representation in atmospheric datasets and numerical weather prediction models, and the development of machine learning approaches for detecting or predicting lake-effect snowfall events.

EARTH SCIENCE > ATMOSPHERE > PRECIPITATION > SOLID↗