Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data formats”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12

Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

The Adaptable I/O System (ADIOS) represents the culmination of substantial investment in Scientific Data Management, and it has demonstrated success for several important extreme-scale science cases. However, looking towards the exascale and beyond, we see the development of yet more stringent data management requirements that require new abstractions. Therefore, there is an opportunity to attempt to connect the traditional realms of HPC I/O optimization with the Database / Data Management community. As such, in this paper we offer some specific examples from our ongoing work in managing data structures, services, and performance at the extreme scale for scientific computing. Using the publish/subscribe model afforded by ADIOS, we demonstrate a set of services that connect data format, metadata, queries, data reduction, and high-performance delivery. The resulting publish/subscribe framework facilitates connection to on-line workflow systems to enable the dynamic capabilities that will be required for exascale science.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Streamlining and Transparently Automating Complex Waste Determinations at Los Alamos National Laboratory - 17286

Environmental programs at Los Alamos National Laboratory (LANL) generate a large amount of sampling activities that can result in waste that needs to be characterized and properly disposed. Since 2011, sampling results are managed in the cloud-based Environmental Information Management (EIM) database, which contains over 16 million current and historical data records. In many cases, the database's analytical records are used to characterize and dispose waste from environmental operations. Three factors contribute to streamlining of the waste disposition operations: (1) if possible, use of the existing analytical data for waste characterization to avoid additional laboratory analysis; (2) reduction in time spent on data evaluation for various waste acceptance criteria; and (3) choice of the proper and least expensive disposal alternative. Use of the existing analytical data, when allowed by law, creates some paradoxical problems: when chemical analysis is requested with the purpose of characterizing waste, it creates only 100 to 400 analytical records. However, use of exiting analytical data at LANL may lead to evaluation of tens of thousands of analytical records, with many records for each single analyte. This occurs because data were collected for environmental characterization and not waste characterization. To streamline data analysis for waste disposition using existing data, LANL has created a Waste Data Summary (WDS) module in the site's EIM database. Recognizing that waste determination is performed by groups outside the environmental program at LANL, the WDS allows for logically guided data selection for a specific waste determination (solids, liquids, radiological, etc.) and reduces potentially thousands of rows of data into simple formatted data reports that are easy to comprehend and process. This allows the waste evaluators to review data using multiple approaches to make their own determination on the most appropriate disposal method. There are four basic principles of WDS operation: (1) The highest detected analytical result is evaluated for the selected samples (or the lowest undetected result if there are no detected results); (2) All approved disposal pathways are evaluated; (3) Any failures of the Waste Acceptance Criteria are visibly highlighted to focus the attention of the evaluator; and (4) The decision of the disposal path is left to the trained waste evaluator (WDS does not choose or suggest any disposal paths). The summary results are presented as the set of short data reports. These reports can be divided into two groups: general comparison reports and disposal pathway specific forms. The general comparison forms show all Resource Conservation and Recovery Act (RCRA) analytes with pass/fail evaluation and with preference for toxicity characteristic leaching procedure (TCLP) results (or 20x rule) for solid samples, the potential RCRA hazard codes assignment for detected analytes, and all rad results. The detected chemical and radioactivity results are compared to the background values and to Soil Screening and Soil Action limits (SSL/SAL limits). Disposal pathway specific forms follow Waste Acceptance Criteria requirements for all disposal pathways available at LANL. The programming of the WDS tries to maximize existing data by analyte substitutions (like using gross alpha for necessary alpha emitting isotopes) or by making calculations when analytical results do not agree with legal requirements (like total chlordane in the RCRA regulations, which is most often analyzed and reported as individual isomers). All forms may be annotated by the waste evaluator to provide additional information to support decision making. Upon final decision of a disposal path, the complete dataset used in the evaluation and the WDS forms with annotations is 'frozen' as an un-editable PDF export to document the data used in the waste disposal determination. The streamlined process results in less work for waste evaluators and reduces overall labor and analytical costs. By comparison, a full review of a 10,000 record parent dataset may take a waste evaluator over a week to review, whereas with the WDS module, the task can be accomplished in less than one hour with clear, reproducible results. These results can be reviewed and checked at any time by quality assurance personnel, thus ensuring transparency in the decision making process. LANL believes this approach has broad applicability to other sites where waste determination is based on review of existing data collected for more generalized environmental purposes. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Cloud Absorption Radiometer Autonomous Navigation System - CANS

CAR (cloud absorption radiometer) acquires spatial reference data from host aircraft navigation systems. This poses various problems during CAR data reduction, including navigation data format, accuracy of position data, accuracy of airframe inertial data, and navigation data rate. Incorporating its own navigation system, which included GPS (Global Positioning System), roll axis inertia and rates, and three axis acceleration, CANS expedites data reduction and increases the accuracy of the CAR end data product. CANS provides a self-contained navigation system for the CAR, using inertial reference and GPS positional information. The intent of the software application was to correct the sensor with respect to aircraft roll in real time based upon inputs from a precision navigation sensor. In addition, the navigation information (including GPS position), attitude data, and sensor position details are all streamed to a remote system for recording and later analysis. CANS comprises a commercially available inertial navigation system with integral GPS capability (Attitude Heading Reference System AHRS) integrated into the CAR support structure and data system. The unit is attached to the bottom of the tripod support structure. The related GPS antenna is located on the P-3 radome immediately above the CAR. The AHRS unit provides a RS-232 data stream containing global position and inertial attitude and velocity data to the CAR, which is recorded concurrently with the CAR data. This independence from aircraft navigation input provides for position and inertial state data that accounts for very small changes in aircraft attitude and position, sensed at the CAR location as opposed to aircraft state sensors typically installed close to the aircraft center of gravity. More accurate positional data enables quicker CAR data reduction with better resolution. The CANS software operates in two modes: initialization/calibration and operational. In the initialization/calibration mode, the software aligns the precision navigation sensors and initializes the communications interfaces with the sensor and the remote computing system. It also monitors the navigation data state for quality and ensures that the system maintains the required fidelity for attitude and positional information. In the operational mode, the software runs at 12.5 Hz and gathers the required navigation/attitude data, computes the required sensor correction values, and then commands the sensor to the required roll correction. In this manner, the sensor will stay very near to vertical at all times, greatly improving the resulting collected data and imagery. CANS greatly improves quality of resulting imagery and data collected. In addition, the software component of the system outputs a concisely formatted, high-speed data stream that can be used for further science data processing. This precision, time-stamped data also can benefit other instruments on the same aircraft platform by providing extra information from the mission flight.

Kahle, Duncan↗

Comparison of methods for geologic storage of carbon dioxide in saline formations

Preliminary estimates of CO{sub 2} storage potential in geologic formations provide critical information related to Carbon Capture, Utilization, and Storage (CCUS) technologies to mitigate CO{sub 2} emissions. Currently multiple methods to estimate CO{sub 2} storage and multiple storage estimates for saline formations have been published, leading to potential uncertainty when comparing estimates from different studies. In this work, carbon dioxide storage estimates are compared by applying several commonly used methods to general saline formation data sets to assess the impact that the choice of method has on the results. Specifically, six CO{sub 2} storage methods were applied to thirteen saline formation data sets which were based on formations across the United States with adaptations to provide the geologic inputs required by each method. Methods applied include those by (1) international efforts – the Carbon Sequestration Leadership Forum (Bachu et al., 2007); (2) United States government agencies – U.S. Department of Energy – National Energy Technology Laboratory (US-DOE-NETL, 2012) and United States Geological Survey (Brennan et al., 2010); and (3) the peer-reviewed scientific community – Szulczewski et al. (2012) and Zhou et al. (2008). A statistical analysis of the estimates generated by multiple methods revealed that assessments of CO{sub 2} storage potential made at the prospective level were often statistically indistinguishable from each other, implying that the differences in methodologies are small with respect to the uncertainties in the geologic properties of storage rock in the absence of detailed site-specific characterization.

54 ENVIRONMENTAL SCIENCES↗

Natural Language Processing for Extracting Rich Disease Data Aligned To Satellite Meteorological Data

Global climate change is redefining our understanding of how diseases spread. In Sri Lanka, vector-borne diseases such as dengue fever, encephalitis, and leptospirosis historically surged during the monsoon seasons when temperatures were high enough for mosquito eggs to hatch. Unfortunately, due to rising temperatures and more erratic rainfall patterns, mosquito eggs can now hatch year-round and are increasingly unpredictable, leading to an alarmingly increasing number of hospitalizations and deaths. More data is needed to adapt our response to these diseases in an increasingly warmer world. In the contemporary landscape, a wealth of disease information is available, yet accessibility remains limited due to unstructured data formats such as PDFs. Therefore, converting unstructured disease reports into structured formats is necessary for effectively leveraging data. This paper introduces a comprehensive framework for collecting unstructured disease reports and transforming them into analyzable formats. By creating separate models tailored to each data format, we can ensure accuracy compared to general models. These straightforward models enhance accessibility and empower other researchers to use our tools. The returned structured data can then be harnessed for analysis, statistical purposes, and informing evidence-based public health interventions, thus facilitating more informed decision-making in healthcare. We deploy this framework to produce geospatial data for Sri Lanka and Brazil for many different conditions and align these data with satellite environmental data, providing for the first time a structured, aligned powerful dataset for disease modeling.

Open Source Open Science↗

NASA's Next Generation of Atmospheric Data Science

The Multi-Angle Imager for Aerosols (MAIA) and the Tropospheric Emission: Monitoring of Pollution(TEMPO) are NASA’s next-generation satellite missions for air quality monitoring. These missions will produce high-quality, high-resolution air quality data to support cross-displinary research. The MAIA mission is collaborating with health science researchers and epidemiologists to study the impacts of air quality on health outcomes. TEMPO aims to improve our understanding of tropospheric air pollution chemistry and our ability to make predictions about air quality and climate forcing. TEMPO will offer hourly measurements of tropospheric ozone, aerosols, and clouds focused on North America at high-spatial resolution, while MAIA will produce high-resolution measurements of speciated particulate matter targeting densely populated cities around the globe. Data from these missions will help improve our understanding of the sources, types, and interactions among the aerosols and trace gases that are polluting Earth’s atmosphere, as well as our understanding of the impact of air pollution on pollution on a wide range of important areas including human health, agriculture, weather, and climate change. The challenges of cross-disciplinary research, computationally expensive multi-variate analyses, and high-resolution data at both local and global scales are driving substantial changes across all of NASA’s Distributed Active Archive Centers (DAACs). High resolution data at scales such these requires a new approach to data ingest, archive, and publication. Like other NASA DAACs, the Atmospheric Science Data Center (ASDC), the DAAC that will be responsible for publishing MAIA and TEMPO data products has historically archived and distributed data on premise. DAACs of the future will archive and distribute data in the cloud, enabling them to remake themselves as research-focused data centers that will support on-demand, data-intensive computations for highly accurate retrospective analyses and predictions. Under the new paradigm, data formats and metadata must support on-demand spatial and temporal sub-setting, as well as other data transformation services such as re-gridding and re-sampling. This presentation will discuss work being done to address data formatting and metadata requirements in this dynamic new environment. In addition to the changes in data stewardship practices at the ASDC, the increased focus on supporting scientific research is driving changes in the relationship between DAACs and researchers. While the ASDC will continue to provide first rate data management and stewardship, it is increasingly focused on serving as a partner not only to the science teams that gather and produce the data it publishes, but to the researchers that use that data.

Beth Huffer↗

MIRAGE: The data acquisition, analysis, and display system

Developed for the NASA Johnson Space Center and Life Sciences Directorate by GE Government Services, the Microcomputer Integrated Real-time Acquisition Ground Equipment (MIRAGE) system is a portable ground support system for Spacelab life sciences experiments. The MIRAGE system can acquire digital or analog data. Digital data may be NRZ-formatted telemetry packets of packets from a network interface. Analog signal are digitized and stored in experimental packet format. Data packets from any acquisition source are archived to a disk as they are received. Meta-parameters are generated from the data packet parameters by applying mathematical and logical operators. Parameters are displayed in text and graphical form or output to analog devices. Experiment data packets may be retransmitted through the network interface. Data stream definition, experiment parameter format, parameter displays, and other variables are configured using spreadsheet database. A database can be developed to support virtually any data packet format. The user interface provides menu- and icon-driven program control. The MIRAGE system can be integrated with other workstations to perform a variety of functions. The generic capabilities, adaptability and ease of use make the MIRAGE a cost-effective solution to many experimental data processing requirements.

Rosser, Robert S.↗

Methods of downloading to user institutions

The Pilot Climate Data System (PCDS) not only supports the ability to output data in a uniform structure via the Common Data Format (CDF, but also supports the ability to provide data in native format for any data set supported by the PCDS. Methods are discussed for acquiring data in either format from the PCDS for further work at remote sites. Four levels of remote utilization are defined, based on the extent of offloading the National Space Science Data Center (NSSDC) computer and local PCDS processing. Characteristics of each level are thoroughly explained, including details of information and data transfers, downloading, uploading, and offloading of the NSSDC computer. Only the levels themselves are specified here. The first level defined is that of a network-based distributed PCDS. A subset of the PCDS software is ported to another VAX and made available on a network (i.e., SPAN) node. There is no subset of the PCDS at the second level, but it is also network based. Non-network utilization of the PCDS, requiring dial-up log on, is denoted as a third level. Finally, at the fourth level, personal computer utilization of the PCDS through dial-up log on with proper terminal emulation is defined.

Treinish, L.↗

Total Ozone Mapping Spectrometer (TOMS) Level-3 Data Products User's Guide

Data from the TOMS series of instruments span the time period from November 1978, through the present with about a one and a-half year gap from January 1994 through July 1996. A set of four parameters derived from the TOMS measurements have been archived in the form of daily global maps or Level-3 data products. These products are total column ozone, effective surface reflectivity, aerosol index, and erythermal ultraviolet estimated at the Earth surface. A common fixed grid of I degree latitude by 1.25 degree longitude cells over the entire globe is provided daily for each parameter. These data are archived at the Goddard Space Flight Center Distributed Active Archive Center (DAAQ in Hierarchical Data Format (HDF). They are also available in a character format through the TOMS web site at http://toms.gsfc.nasa.gov. The derivations of the parameters, the mapping algorithm, and the data formats are described. The trend uncertainty for individual TOMS instruments is about 1% decade, but additional uncertainty exists in the combined data record due to uncertainty in the relative calibrations of the various TOMS.

McPeters, Richard D.↗

IVS Working Group 4: VLBI Data Structures

In 2007 the IVS Directing Board established IVS Working Group 4 on VLBI Data Structures. This note discusses the current VLBI data format, goals for a new format, the history and formation of the Working Group, and a timeline for the development of a new VLBI data format.

Gipson, John↗

New SPDF Directions and Evolving Services Supporting Heliophysics Research

The next advances in Heliophysics science and its paradigm of a Great Observatory require an increasingly integrated and transparent data environment, where data can be easily accessed and used across the boundaries of both missions and traditional disciplines. The Space Physics Data Facility (SPDF) project includes uniquely important multi-mission data services with current data from most operating space physics missions. This paper reviews the capabilities of key services now available and the directions in which they are expected to evolve to enable future multi-mission correlative research. The Coordinated Data Analysis Web (CDAWeb) and Satellite Situation Center Web (SSCWeb), critically supported by the Common Data Format (CDF) effort and supplemented by more focused science services such as OMNIWeb and technical services such as data format translations are important operational capabilities serving the international community today (and cited last year by 20% of the papers published in JGR Space Physics). These services continue to add data from most current missions as SPDF works with new missions such as THEMIS to help enable their unique science goals and the meaningful sharing of their data in a multi-mission correlative context. Recent enhancements to CDF, our 3D Java interactive orbit viewer (TIPSOD), the CDAWeb Plus system, increasing automation of data service population, the new folding of the VSPO effort into SPDF and our continuing thrust towards fully-functional web services APIs to allow ready invocation from distributed external middleware and clients will be shown.

McGuire, Robert E.↗

Ka-Band ARM Zenith Radar Corrections (KAZRCOR, KAZRCFRCOR) Value-Added Products

The Ka-Band Atmospheric Radiation Measurement (ARM) Zenith Radar Corrections (KAZRCOR) and KA-Band ARM Zenith Radar CF-Radial, Corrected (KAZRCFRCOR) value-added products (VAPs) perform several corrections to the ingested KAZR moments and also create a significant detection mask for each radar mode. The VAPs compute gaseous attenuation as a function of time and radial distance from the radar antenna, based on ambient meteorological observations, and correct observed reflectivities for that effect. Mean Doppler velocities are dealiased to correct velocities whose magnitudes exceed the radar’s Nyquist velocity. Input KAZR data fields are passed through to the KAZRCOR or KAZRCFRCOR output files, in their native time and range coordinates. Complementary corrected reflectivity and velocity fields are provided, along with a mask of significant detections and a number of data quality flags. This report covers the KAZRCOR VAP as applied to the original KAZR radars and the upgraded KAZR2 radars. Originally, prior to late 2019, there were two separate code bases for the different radar versions. Following the harmonization of KAZR and KAZR2 data formats in 2019, only a single code base is required. The new combined KAZR and KAZR2 code base is called the KAZRCFRCOR VAP. The ‘cfr portion of the VAP name refers to the Radial Climate and Forecasting data format standards that are used in these data sets. Throughout this report, references to ‘KAZRCOR’ should be taken to apply to the ‘KAZRCFRCOR’ VAP as well unless there is an explicit statement to the contrary.

54 ENVIRONMENTAL SCIENCES↗

The ATLAS Trigger Simulation with Legacy Software

Physics analyses at the LHC require accurate simulations of the detector response and the event selection processes, generally done with the most recent software releases. The trigger response simulation is crucial for determination of overall selection efficiencies and signal sensitivities and should be done with the same software release with which data were recorded. This requires potentially running with software dating many years back, the so-called legacy software, in which algorithms and configuration may differ from their current implementation. Therefore having a strategy for running legacy software in a modern environment becomes essential when data simulated for past years start to present a sizeable fraction of the total. The requirements and possibilities for such a simulation scheme within the ATLAS software framework were examined and a proof-of-concept simulation chain has been successfully implemented. One of the greatest challenges was the choice of a data format which promises long term compatibility with old and new software releases. Over the time periods envisaged, data format incompatibilities are also likely to emerge in databases and other external support services. Software availability may become an issue, when e.g. the support for the underlying operating system might stop. The encountered problems and developed solutions will be presented, and proposals for future development will be discussed. Some ideas reach beyond the retrospective trigger simulation scheme in ATLAS as they also touch more generally aspects of data preservation.

97 MATHEMATICS AND COMPUTING↗

Historic Source Inspection Metadata

Although this data dictionary describes the raw data format, it is relevant only for data entered in 2017 and later. Entries in this date range represent formatted, standardized data that have been verified against original, raw data files by the database manager. She took ownership of the database in September 2022 and was able to standardize entries by comparing against raw data files back to 2017. Furthermore, for comparison of historic pre-source inspection data to new pre-source data supplied by the vendor and intended for action limit analysis purposes, this time period is of sufficient length.

97 MATHEMATICS AND COMPUTING↗

Mass Spectrometry Imaging of Biological Tissue: An Approach for Multicenter Studies

Mass spectrometry imaging has become a popular tool for probing the chemical complexity of biological surfaces. This led to the development of a wide range of instrumentation and preparation protocols. It is thus desirable to evaluate and compare the data output from different methodologies and mass spectrometers. Here, we present an approach for the comparison of mass spectrometry imaging data from different laboratories (often referred to as multicenter studies). This is exemplified by the analysis of mouse brain sections in five laboratories in Europe and the USA. The instrumentation includes matrix-assisted laser desorption/ionization (MALDI)-time-of-flight (TOF), MALDI-QTOF, MALDIFourier transform ion cyclotron resonance (FTICR), atmospheric-pressure (AP)-MALDI-Orbitrap, and cluster TOF-secondary ion mass spectrometry (SIMS). Experimental parameters such as measurement speed, imaging bin width, and mass spectrometric parameters are discussed. All datasets were converted to the standard data format imzML and displayed in a common open-source software with identical parameters for visualization, which facilitates direct comparison of MS images. The imzML conversion also allowed exchange of fully functional MS imaging datasets between the different laboratories. The experiments ranged from overview measurements of the full mouse brain to detailed analysis of smaller features (depending on spatial resolution settings), but common histological features such as the corpus callosum were visible in all measurements. High spatial resolution measurements of AP-MALDI-Orbitrap and TOF-SIMS showed comparable structures in the low-micrometer range. We discuss general considerations for planning and performing multicenter studies in mass spectrometry imaging. This includes details on the selection, distribution, and preparation of tissue samples as well as on data handling. Such multicenter studies in combination with ongoing activities for reporting guidelines, a common data format (imzML) and a public data repository can contribute to more reliability and transparency of MS imaging studies.

Environmental Molecular Sciences Laboratory↗

Raw tensiometer soil matric potential data, from Oct 2017-Jun 2020 in Manaus, Brazil

This data package contains raw output from two Campbell Scientific CR1000 data loggers each connected to three UMS T4e tensiometer sensors in Manaus, Brazil. Tensiometers at each data logger location were installed to depths of 10 cm, 20 cm and 50 cm below the ground surface for the purpose of measuring soil matric potential at each depth. “Station 3” tensiometers were installed at approximately 2°36'31.68"S, 60°12'34.70"W. “Station 4” tensiometers were installed at approximately 2°36'32.46"S, 60°12'34.92"W. The files “tensio3.edit.202109.csv” and “tensio4.edit.202109.csv” contain raw data (with invalid data removed). A text editor will be required to view the raw files in .csv (comma-separated values) format. Data processing was limited to: 1) Editing column headers for clarity; 2) Applying a correction that converts voltage read by the data loggers to pressure units (kPa); 3) removing all data greater than 50 kPa and less than -115 kPa; 4) Adding columns entitled “Caution.10cm”, “Caution.20cm” and “Caution.50cm” that contains values of either “NA” or “Caution”, the latter of which is applied when data exceeds the manufacturer reported data validity range of less than -85 kPa to suggest use of these data with caution; 5) Adding columns entitled “Invalid.10cm”, “Invalid.20cm”, and “Invalid.50cm” that contains values of either “NA” or “Invalid”, the latter of which is applied when data accuracy is deemed compromised due to the introduction of air bubbles into the instrument. Data marked “Invalid” may still indicate general trends of wet/high matric potential and dry/low matric potential, but values are inaccurate and dry/low matric potential values are likely not low enough. Wet/high matric potential values are likely to be more accurate than dry/low matric potential values; 6) Data format changed from .dat to .csv. This research was performed as a part of the NGEE Tropics project, which aims to advance model predictions of tropical forest carbon cycle responses to a changing climate over the 21st Century.

54 ENVIRONMENTAL SCIENCES↗

SPAR improved structural-fluid dynamic analysis capability

The results of a study whose objective was to improve the operation of the SPAR computer code by improving efficiency, user features, and documentation is presented. Additional capability was added to the SPAR arithmetic utility system, including trigonometric functions, numerical integration, interpolation, and matrix combinations. Improvements were made in the EIG processor. A processor was created to compute and store principal stresses in table-format data sets. An additional capability was developed and incorporated into the plot processor which permits plotting directly from table-format data sets. Documentation of all these features is provided in the form of updates to the SPAR users manual.

Pearson, M. L.↗