Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “scientific data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 955 records · Page 53
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Enabling Scalable and Extensible Memory-mapped Datastores in Userspace

Exascale workloads are expected to incorporate data-intensive processing in close coordination with traditional physics simulations. These emerging scientific, data-analytics and machine learning applications need to access a wide variety of datastores in flat files and structured databases. Programmer productivity is greatly enhanced by mapping datastores into the application process's virtual memory space to provide a unified “in-memory” interface. Currently, memory mapping is provided by system software primarily designed for generality and reliability. However, scalability at high concurrency is a formidable challenge on exascale systems. Also, there is a need for extensibility to support new datastores potentially requiring HPC data transfer services. In this article, we present UMap , a scalable and extensible userspace service for memory-mapping datastores. Furthermore, through decoupled queue management, concurrency aware adaptation, and dynamic load balancing, UMap enables application performance to scale even at high concurrency. We evaluate UMap in data-intensive applications, including sorting, graph traversal, database operations, and metagenomic analytics. Our results show that UMap as a userspace service outperforms an optimized kernel-based service across a wide range of intra-node concurrency by 1.22-1.9 × . We performed two case studies to demonstrate UMap 's extensibility. First, a new datastore residing in remote memory is incorporated into UMap as an application-specific plugin. Second, we present a persistent memory allocator Metall built atop UMap for unified storage/memory.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1982

A plan for the characterization, calibration, and evaluation of LAPR-2

A new airborne Linear Array Pushbroom Radiometer (LAPR-II) was built. LAPR-II will use linear arrays of silicon detectors to acquire four channels of digital image data for spectral bands within the visible and near infrared portions of the spectrum (0.4 - 1.0 micrometers). The data will be quantized to 10 bits, and spectral filters for each channel will be changeable in flight. The instrument will initially be flown aboard a NASA/Wallops' aircraft, and off nadir pointing of LAPR-II will be possible. Together, the instrument and its platform will provide a flexible readily available source of digital image data for scientific experiments. If LAPR-II is to serve as a precise scientific instrument, the instrument's characteristics must be quantitatively described and the data must be calibrated with respect to absolute radiometric units. The LAPR-II is described and the work required to characterize the instrument's spectral response, radiometric response, and spatial resolution and to calibrate the response from the many detectors per array is outlined.

Ilrons, J. R.↗
NASA NTRS1990

Planetary astronomy of Mars. Mars data reduction

Grant #NAGW-1408 was specifically awarded to obtain telescopic visible and near-IR spectral imaging of Mars during the 1988 apparition (9/28/88). The observing program was highly successful producing approximately 2 Gbytes of data, but was only funded for one year and virtually all of the funds were spent in data acquisition. The follow-up grant was funded the following year for reduction of these data into a scientifically productive form, which because of the size and nature of our observations, was a non-trivial task. A more detailed scientific analysis of these data (fully reduced) is in progress now and will take a number of years. Extended geologic analyses of the astronomical data are being funded by the NASA Planetary Geology and Geophysics program. The objective was to produce detailed reflectance spectra for contiguous, spatially resolved surface elements covering most of the planet (about +50 degrees to -90 degrees latitude, all longitudes). A total of 6 observing runs, of 3-4 days duration each, were conducted on the University of Arizona's 1.5m telescope on Mt. Bigelow. We present a sketch map showing the approximate total extent of our spectral image coverage. Nearly all of Mars south of 40 degrees N was observed at least once. About half of the area shown was observed multiple times. South of 65 degrees S, including the south polar cap, our coverage is heavily redundant. The first run was conducted June 29 - July 1, 1988 (all dates are UT) to serve as a baseline prior to possible dust storm activity on Mars. The other observing runs were closer to opposition: Sept. 3-6, 13-15, 24-26, and Oct. 5-7 and 16-18. The September and October observations were scheduled to provide maximum longitudinal coverage. This was also intended to provide a balance between surface observations and observations of predicted dust storm activity. No global dust storm developed in 1988, so we have a large volume of data for the surface of Mars.

Singer, Robert B.↗
NASA NTRS1992

Non-Scanning Radiometer Results for Earth Radiation Budget Investigations

The Earth Radiation Budget Experiment (ERBE) included non-scanning radiometers (Luther, 1986) flown aboard a dedicated mission of Earth Radiation Budget Satellite, and the NOAA-9 and -10 operational meteorological spacecraft (Barkstrom and Smith, 1986). The radiometers first began providing Earth radiation budget data in November 1984 and have remained operational, providing a record of nearly 8 years of data to date for researchers. Although they do not produce measurements with the resolution given by the scanning radiometers, the results from the non-scanning radiometers are extremely useful for climate research involving long-term radiation data sets. This paper discusses the non-scanning radiometers, their stability, the method of analyzing the data, and brief scientific results from the data.

Smith, G. Louis↗
NASA NTRSDate not supplied

Airborne for All Engaging the ESIP Community to Explore Findings from a Recent Airborne and Field Data Workshop and Identify Synergies Across Organizations

The highly diverse, heterogeneous, and cross-disciplinary nature of airborne Earth observation provides unique challenges for those who want to use the data for scientific research and applications. NASA recently held its first Airborne and Field data workshop as a collaborative, open forum for users and producers of airborne Earth observations to discuss their experiences and concerns. During the two day event, NASA collected feedback from attendees about what NASA is doing well and what needs are still unmet in order to determine how NASA can better help the community make use of airborne and field data. In this session, we will report on the workshop findings. We will discuss the needs discovered from the workshop and we look for community feedback about how NASA can best address community concerns and work with other organizations that collect airborne data.

Sara Lubkin↗
NASA NTRSDate not supplied

NASA’s Commercial Smallsat Data Acquisition (CSDA) Program Data System

The National Aeronautic and Space Administration (NASA) Commercial Smallsat Data Acquisition (CSDA) Program was established to identify, evaluate, and acquire data from commercial satellite companies that support and complement NASA’s Earth sciences missions and research goals. Since becoming a sustained program in 2020, CSDA has been developing a data system to provide scalable, efficient, continuous, and repeatable data management processes for all commercial data acquired by NASA. This includes providing access to commercial data to approved science investigators through NASA developed and vendor operated interfaces. As the commercial data user community and volume of data acquired by CSDA has grown, the team managing these data has adapted technologies and services to better address the data and information search, discovery, and access needs of the science user community. This presentation will provide an overview of CSDA Program data system activities including data availability, development of CSDA user interfaces, challenges in managing large-volume diverse datasets, and long-term data preservation activities to retain data for scientific reproducibility.

data management↗
DOE OSTI2026

Machine Learning on Heterogeneous, Edge, and Quantum Hardware for Particle Physics (ML-HEQUPP)

The next generation of particle physics experiments will face a new era of challenges in data acquisition, due to unprecedented data rates and volumes along with extreme environments and operational constraints. Harnessing this data for scientific discovery demands real-time inference and decision-making, intelligent data reduction, and efficient processing architectures beyond current capabilities. Crucial to the success of this experimental paradigm are several emerging technologies, such as artificial intelligence and machine learning (AI/ML) and silicon microelectronics, and the advent of quantum algorithms and processing. Their intersection includes areas of research such as low-power and low-latency devices for edge computing, heterogeneous accelerator systems, reconfigurable hardware, novel codesign and synthesis strategies, readout for cryogenic or high-radiation environments, and analog computing. This white paper presents a community-driven vision to identify and prioritize research and development opportunities in hardware-based ML systems and corresponding physics applications, contributing towards a successful transition to the new data frontier of fundamental science.

Gonski, Julia [SLAC]↗
NASA NTRS1986

Off-line processing of ERS-1 synthetic aperture radar data with high precision and high throughput

The first European remote sensing satellite ERS-1 will be launched by the European Space Agency (ESA) in 1989. The expected lifetime is two to three years. The spacecraft sensors will primarily support ocean investigations and to a limited extent also land applications. Prime sensor is the Active Microwave Instrumentation (AMI) operating in C-Band either as Synthetic Aperture Radar (SAR) or as Wave-Scatterometer and simultaneously as Wind-Scatterometer. In Europe there will be two distinct types of processing for ERS-1 SAR data, Fast Delivery Processing and Precision Processing. Fast Delivery Proceessing will be carried out at the ground stations and up to three Fast Delivery products per pass will be delivered to end users via satellite within three hours after data acquisition. Precision Processing will be carried out in delayed time and products will not be generated until several days or weeks after data acquisition. However, a wide range of products will be generated by several Processing and Archiving Facilities (PAF) in a joint effort coordinated by ESA. The German Remote Sensing Data Center (Deutsches Fernerkundungsdatenzentrum DFD) will develop and operate one of these facilities. The related activities include the acquisition, processing and evaluation of such data for scientific, public and commercial users. Based on this experience the German Remote Sensing Data Center is presently performing a Phase-B study regarding the development of a SAR processor for ERS-1. The conceptual design of this processing facility is briefly outlined.

Gredel, J.↗
NASA NTRS2007

Processing Raw HST Data With Up-to-Date Calibration Data

On-the-Fly Reprocessing (OTFR) is a collection of data-processing routines that work within the context of the Hubble Space Telescope (HST) pipeline data-flow system. The purpose served by OTFR is to generate, on demand, scientifically useful data products from raw HST data stored in an archive. First, on the basis of the requested final data products, OTFR retrieves the corresponding sets of raw data from the archives. Next, OTFR processes the raw data sets to remove artifacts and to establish proper header and other template information. Finally, the calibration routines appropriate to the specific data sets are invoked to produce the requested data products, and the data products are released to an archive distribution system for transmission to the requesting party. OTFR offers two notable advantages: (1) Inasmuch as calibrated data occupy about 8 times as much storage space as do raw data, by obviating storage of calibrated data, OTFR reduces the storage capacity needed by the archive; and (2) the calibration routines can be updated to give requesters the benefit of the most recent calibrations.

Miller, Warren↗
DOE OSTI2022

EdgeAI: Machine learning via direct attached accelerator for streaming data processing at high shot rate x-ray free-electron lasers

We present a case for low batch-size inference with the potential for adaptive training of a lean encoder model. We do so in the context of a paradigmatic example of machine learning as applied in data acquisition at high data velocity scientific user facilities such as the Linac Coherent Light Source-II x-ray Free-Electron Laser. We discuss how a low-latency inference model operating at the data acquisition edge can capitalize on the naturally stochastic nature of such sources. We simulate the method of attosecond angular streaking to produce representative results whereby simulated input data reproduce high-resolution ground truth probability distributions. By minimizing the mean-squared error between the decoded output of the latent representation and the ground truth distributions, we ensure that the encoding layers and resulting latent representation maintains full fidelity for any downstream task, be it classification or regression. We present throughput results for data-parallel inference of various batch sizes, some with throughput exceeding 100 k images per second. We also show in situ training below 10 s per epoch for the full encoder–decoder model as would be relevant for streaming and adaptive real-time data production at our nation’s scientific light sources.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1984

A calibration adjustment technique combining ERB parameters from different remote sensing platforms into a long-term data set

Earth-radiation-budget (ERB) experiments on board the Nimbus-6 (ERB 6) and Nimbus-7 (ERB 7) spacecraft have measured wide-field-of-view total (0.2 to 50 micron), shortwave (0.2 to 3.8 micron), and near-infrared (0.7 to 2.8 micron) terrestrial irradiances for a joint lifetime of over 8 years. Though the spectral characteristics of both experiments are nearly identical, instrument degradation and altitude differences introduce discrepancies between the two data sets. ERB parameters from these two observing platforms may be combined into a scientifically meaningful data set only after these discrepancies are eliminated. To facilitate the creation of a long-term ERB data set, comparisons of the ERB 6 experiment irradiances with, and calibration adjustments with respect to, the corresponding ERB 7 irradiances have been performed. Two calibration methods were developed and applied to the irradiance data from 28 pairs of collocated orbits. The differential effects of altitude, illumination, albedo, and scene inhomogeneities were applied. The result is a set of calibration adjustments that adjust the ERB 6 data to match the ERB 7 data.

Ardanuy, P. E.↗
NASA NTRS2016

Geocam Space: Enhancing Handheld Digital Camera Imagery from the International Space Station for Research and Applications

Handheld astronaut photography of the Earth has been collected from the International Space Station (ISS) since 2000, making it the most temporally extensive remotely sensed dataset from this unique Low Earth orbital platform. Exclusive use of digital handheld cameras to perform Earth observations from the ISS began in 2004. Nadir viewing imagery is constrained by the inclined equatorial orbit of the ISS to between 51.6 degrees North and South latitude, however numerous oblique images of land surfaces above these latitudes are included in the dataset. While unmodified commercial off-the-shelf digital cameras provide only visible wavelength, three-band spectral information of limited quality current cameras used with long (400+ mm) lenses can obtain high quality spatial information approaching 2 meters/ground pixel resolution. The dataset is freely available online at the Gateway to Astronaut Photography of Earth site (http://eol.jsc.nasa.gov), and now comprises over 2 million images. Despite this extensive image catalog, use of the data for scientific research, disaster response, commercial applications and visualizations is minimal in comparison to other data collected from free-flying satellite platforms such as Landsat, Worldview, etc. This is due primarily to the lack of fully-georeferenced data products - while current digital cameras typically have integrated GPS, this does not function in the Low Earth Orbit environment. The Earth Science and Remote Sensing (ESRS) Unit at NASA Johnson Space Center provides training in Earth Science topics to ISS crews, performs daily operations and Earth observation target delivery to crews through the Crew Earth Observations (CEO) Facility on board ISS, and also catalogs digital handheld imagery acquired from orbit by manually adding descriptive metadata and determining an image geographic centerpoint using visual feature matching with other georeferenced data, e.g. Landsat, Google Earth, etc. The lack of full geolocation information native to the data makes it difficult to integrate astronaut photographs with other georeferenced data to facilitate quantitative analysis such as urban land cover/land use classification, change detection, or geologic mapping. The manual determination of image centerpoints is both time and labor-intensive, leading to delays in releasing geolocated and cataloged data to the public, such as the timely use of data for disaster response. The GeoCam Space project was funded by the ISS Program in 2015 to develop an on-orbit hardware and ground-based software system for increasing the efficiency of geolocating astronaut photographs from the ISS (Fig. 1). The Intelligent Robotics Group at NASA Ames Research Center leads the development of both the ground and on-orbit systems in collaboration with the ESRS Unit. The hardware component consists of modified smartphone elements including cameras, central processing unit, wireless Ethernet, and an inertial measurement unit (gyroscopes/accelerometers/magnetometers) reconfigured into a compact unit that attaches to the base of the current Nikon D4 camera - and its replacement, the Nikon D5 - and connects using the standard Nikon peripheral connector or USB port. This provides secondary, side and downward facing cameras perpendicular to the primary camera pointing direction. The secondary cameras observe calibration targets with known internal X, Y, and Z position affixed to the interior of the ISS to determine the camera pose corresponding to each image frame. This information is recorded by the GeoCam Space unit and indexed for correlation to the camera time recorded for each image frame. Data - image, EXIF header, and camera pose information - is transmitted to the ground software system (GeoRef) using the established Ku-band USOS downlink system. Following integration on the ground, the camera pose information provides an initial geolocation estimate for the individual film frame. This new capability represents a significant advance in geolocation from the manual feature-matching approach for both nadir and off-nadir viewing imagery. With the initial geolocation estimate, full georeferencing of an image is completed using the rapid tie-pointing interface in GeoRef, and the resulting data is added to the Gateway to Astronaut Photography of Earth online database in both Geotiff and Keyhole Markup Language (kml) formats. The integration of the GeoRef software component of Geocam Space into the CEO image cataloging workflow is complete, and disaster response imagery acquired by the ISS crew is now fully georeferenced as a standard data product. The on-orbit hardware component (GeoSens) is in final prototyping phase, and is on-schedule for launch to the ISS in late 2016. Installation and routine use of the Geocam Space system for handheld digital camera photography from the ISS is expected to significantly improve the usefulness of this unique dataset for a variety of public- and private-sector applications.

Stefanov, William L.↗
DOE OSTI2020

Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

Lassnig, Mario↗
NASA NTRS1975

Trajectory, atmosphere, and wind reconstruction from Viking entry measurements

During Mars entry, each Viking Lander will collect and send to earth large quantities of measurement data, both scientific and engineering in nature. Software has been developed to statistically process this and other data in order to reconstruct, for mission support purposes, the entry trajectory, an 'engineering' atmosphere, and a wind profile. Procedures are described for processing limited amounts of the available data in such a manner that the maximum amount of information can be extracted therefrom, consistent with program design, response requirements, and mission operations constraints. Results of observability studies and simulations indicate that accuracy requirements will be met.

Hopper, F. W.↗
NASA NTRS2003

SOHO Ultraviolet Coronagraph Spectrometer (UVCS) Mission Operations and Data Analysis

The scientific goal of UVCS is to obtain detailed empirical descriptions of the extended solar corona as it evolves over the solar cycle and to use these descriptions to identify and understand the physical processes responsible for coronal heating, solar wind acceleration, coronal mass ejections (CMEs), and the phenomena that establish the plasma properties of the solar wind as measured by 'in situ' solar wind instruments. This report covers the period from 01 February 2002 to 15 February 2003. During that time, UVCS observations have consisted of three types: 1) standard synoptic observations comprising, primarily, the H I Ly alpha line profile and the O VI 103.2 and 103.7 nm intensity over a range of heights from 1.5 to about 3.0 solar radii and covering 360 degrees about the sun, 2) sit and stare watches for CMEs, and 3) special observations designed by the UVCS Lead Observer of the Week for a specific scientific purpose. The special observations are often coordinated with those of other space-based and ground-based instruments and they often are part of SOHO joint observation programs and campaigns. Lead observers have included UVCS Co-Investigators, scientists from the solar physics community and several graduate and undergraduate level students. UVCS has continued to achieve its purpose of using powerful spectroscopic diagnostic techniques to obtain a much more detailed description of coronal structures and dynamic phenomena than existed before the SOHO mission. The new descriptions of coronal mass ejections (CMEs) and coronal structures from UVCS have inspired a large number of theoretical studies aimed at identifying the physical processes responsible for CMEs and solar wind acceleration in coronal holes and streamers. UVCS has proven to be a very stable instrument. Stellar observations have demonstrated its stability. UVCS has required no flight software modifications and all mechanisms are operational. The UVCS O VI Channel with its redundant optical path for wavelengths near H I Ly alpha is capable of observing the entire UVCS wavelength range. Since December 1998, the O VI Channel has been used for all UVCS observations. Although the H I Ly alpha Channel and detector are still operational, increases in the dark count up to about 5 x 10(exp -4) counts/sec/pixel and an increase in high voltage current to within a factor of two of the maximum used in the laboratory before flight led to the decision to not use that detector at the present time. There is no significant decrease in the scientific capability of UVCS owing to the O VI channel redundant optical path. UVCS data, data analysis software, calibration files and the mission log are available from the SOHO archive and SAO. All UVCS data is now available to scientists and the general public via the SOHO Data Archive and SAO within three months of the observations. UVCS has resulted in 46 scientific papers in 2002. There were numerous presentations at scientific meetings. All requests for observation time by qualified outside users have been granted.

Kohl, John L.↗
DOE OSTI2021

Scalable, In-situ Data Clustering Data Analysis for Extreme Scale Scientific Computing (Final Report)

The objective of this project is to address challenges in the design and development of scalable in-situ data clustering and analytics algorithms and software. Our goal is to develop parallel software consisting of a set of spatio-temporal data clustering and anomaly detection functions, both of which are very important for large-scale analysis and have wide applicability for in-situ runs as well as post-processing analysis. Our design principles for in-situ analysis consider the following: (1) identify parts of the computation can be done close to the data within the nodes, while it is still in memory; (2) extract analysis components can (and should) be performed in remote staging and analysis nodes; (3) develop error-bound approximation methods for applications tolerable for small errors; (4) identify the type of derived distributions and statistics, for spatio-temporal data, that can be kept locally in order to both accelerate computations and meet energy constraints in subsequent iterations and phases; (5) use a self-describing data format so that data can be consistent and understood among local storage (memory and SSDs) and at staging and analysis nodes, thereby providing portability and flexibility; (6) develop service-oriented functions that can schedule in-situ and post-hoc analysis tasks based on the dynamic requirements of applications. Our development focus is to produce the parallel data analysis software/library that will be scalable, reusable, extensible, and generic for applications in different disciplines. The software will be able to run in-situ with the simulations as well as post-hoc analysis. This approach will satisfy many synergistic requirements for data intensive applications executed on data coming from instruments and experiments. In particular, the proposed multilevel approach is directly applicable to perform design tradeoffs for running part of the algorithms near the instruments and the rest on remote (analysis) systems.

97 MATHEMATICS AND COMPUTING↗