SEARCH · Engineering Papers

Results for “science data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

All sources NASA NTRS DOE OSTI

NASA NTRSDate not supplied

The NASA SRA Process as it Relates to Open-Source Workflows Developed for GeneLab Data Processing

To release open, standards-compliant processed data sets in the Open Science Data Repository (OSDR), the GeneLab Data Processing team works with the scientific community through the OSDR Analysis Working Groups to design and build open-source data processing pipelines. Once baselined internally, these pipelines are wrapped into workflows and published on the NASA GeneLab Data Processing public GitHub repository along with detailed instructions for installation and use. Each workflow must be approved through NASA's Software Release Authorization (SRA) process prior to publishing. However, the SRA process lacks sufficient documentation and clarity regarding which forms are applicable for new open-source software that utilizes publicly available 3rd party tools, and the SRA process can take several months to complete, making sharing software outside of NASA cumbersome and in contradiction with the concept of Open Science. Furthermore, the SRA process was designed as a one-size fits all approach and thus many of the questions asked are not applicable to our open-source workflows. Here we describe the software provided on the NASA GeneLab Data Processing GitHub repository, summarize our experiences with the SRA process to release these software, and propose a more stream-lined approach for review of open-source projects.

Software Release Authorization↗

NASA NTRSDate not supplied

Open Science in Action: The Role of SWxSOC in Expedited Data Release and Cloud-based Data Processing for Heliophysics Missions

NASA's Space Weather Science Operations Center (SWxSOC) is an effort to develop a multi-mission Science Operations Center for the community and specifically Space Weather missions that require data products to be released consistently and quickly. The SWxSOC is committed to Open Science in its approach to software development and data product releases. We are currently supporting the Heliophysics Environmental and Radiation Measurement Experiment Suite (HERMES) that will fly on the Lunar Gateway and the PADRE Small Sat mission. SWxSOC has established an open-source, reusable solution for transitioning data management from on-premises to the cloud, processing data files, and setting up a cloud-based analysis environment to monitor instrument anomalies. Leveraging Amazon Web Services (AWS) and making open-source tools available on GitHub, HERMES exemplifies NASA's commitment to open-source standardization, showcasing the real-world effectiveness of cloud technology in data processing, analysis, and observability. This enhances current operations and ensures faster, standardized deployments for future missions.

hermes↗

DOE OSTI2017

Protein abundances can distinguish between naturally-occurring and laboratory strains of Yersinia pestis , the causative agent of plague

Adaptive processes in bacterial species can occur rapidly in laboratory culture, leading to genetic divergence between naturally occurring and laboratory-adapted strains. Differentiating wild and closely-related laboratory strains is clearly important for biodefense and bioforensics; however, DNA sequence data alone has thus far not provided a clear signature, perhaps due to lack of understanding of how diverse genome changes lead to adapted phenotypes. Protein abundance profiles from mass spectrometry-based proteomics analyses are a molecular measure of phenotype. Proteomics data contains sufficient information that powerful statistical methods can uncover signatures that distinguish wild strains of Yersinia pestis from laboratory-adapted strains.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRS2004

Initial Processing of Infrared Spectral Data

The Atmospheric Infrared Spectrometer (AIRS) Science Processing System is a collection of computer programs, denoted product generation executives (PGEs), for processing the readings of the AIRS suite of infrared and microwave instruments orbiting the Earth aboard NASA's Aqua spacecraft. Following from level 0 (representing raw AIRS data), the PGEs and their data products are denoted by alphanumeric labels (1A, 1B, and 2) that signify the successive stages of processing. Once level-0 data have been received, the level-1A PGEs begin processing, performing such basic housekeeping tasks as ensuring that all the Level-0 data are present and ordering the data according to observation times. The level-1A PGEs then perform geolocation-refinement calculations and conversions of raw data numbers to engineering units. Finally, the level-1A data are grouped into packages, denoted granules, each of which contain the data from a six-minute observation period. The granules are forwarded, along with calibration data, to the Level-1B PGEs for processing into calibrated, geolocated radiance products. The Level-2 PGEs, which are not yet operational, are intended to process the level-1B data into temperature and humidity profiles, and other geophysical properties.

De Picciotto, Solomon↗

NASA NTRS1984

Geocoded data structures and their applications to Earth science investigations

A geocoded data structure is a means for digitally representing a geographically referenced map or image. The characteristics of representative cellular, linked, and hybrid geocoded data structures are reviewed. The data processing requirements of Earth science projects at the Goddard Space Flight Center and the basic tools of geographic data processing are described. Specific ways that new geocoded data structures can be used to adapt these tools to scientists' needs are presented. These include: expanding analysis and modeling capabilities; simplifying the merging of data sets from diverse sources; and saving computer storage space.

Goldberg, M.↗

NASA NTRS2018

PYSAT: Python Satellite Data Analysis Toolkit

A common problem in space science data analysis is combining complementary data sources that are provided and analyzed in different formats and programming languages. The Python Satellite Data Analysis Toolkit (pysat) addresses this issue by providing an open source toolkit that implements the general process of space science data analysis, from beginning to end, in an instrumentindependent manner. This toolkit uses an Instrument object that enables systematic analysis of science data from a variety of platforms within a single interface. Basic functions such as downloading, loading, and cleaning are included for all supported instruments. Common analysis routines are also included, which are instrument and data source independent. A nanokernel is used to provide instrument independence, it is attached to the Instrument object and mediates the systematic and arbitrary modification of loaded data. Pysat uses the nanokernel to improve the rigor of time series analysis, support onthefly orbit determination, and cleanly span file breaks. Pysat's functions and higherlevel scientific analysis features are validated through the use of unit testing. Further adoption by the community provides a set of scientific results produced by a common core, constituting a distributed heritage that supports the validity of the underlying processing and scientific output. These features are used to demonstrate consistency between derived electron density profiles and measured ion drifts, particularly downward ion drifts in the afternoon hours during extreme solar minimum. Pysat builds upon open source Python software that is freely available and encourages communitydriven development.

Stoneback, R.A.↗

DOE OSTI2024

ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science

ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRS2012

VIIRS On-Orbit Calibration for Ocean Color Data Processing

The NASA VIIRS Ocean Science Team (VOST) has the task of evaluating Suomi NPP VIIRS ocean color data for the continuity of the NASA ocean color climate data records. The generation of science quality ocean color data products requires an instrument calibration that is stable over time. Since the VIIRS NIR Degradation Anomaly directly impacts the bands used for atmospheric correction of the ocean color data (Bands M6 and M7), the VOST has adapted the VIIRS on-orbit calibration approach to meet the ocean science requirements. The solar diffuser calibration time series and the solar diffuser stability monitor time series have been used to derive changes in the instrument response and diffuser reflectance over time for bands M1-M11.

Eplee, Robert E., Jr.↗

NASA NTRS2015

Earth Science Data Fusion with Event Building Approach

Objectives of the NASA Information And Data System (NAIADS) project are to develop a prototype of a conceptually new middleware framework to modernize and significantly improve efficiency of the Earth Science data fusion, big data processing and analytics. The key components of the NAIADS include: Service Oriented Architecture (SOA) multi-lingual framework, multi-sensor coincident data Predictor, fast into-memory data Staging, multi-sensor data-Event Builder, complete data-Event streaming (a work flow with minimized IO), on-line data processing control and analytics services. The NAIADS project is leveraging CLARA framework, developed in Jefferson Lab, and integrated with the ZeroMQ messaging library. The science services are prototyped and incorporated into the system. Merging the SCIAMACHY Level-1 observations and MODIS/Terra Level-2 (Clouds and Aerosols) data products, and ECMWF re- analysis will be used for NAIADS demonstration and performance tests in compute Cloud and Cluster environments.

Lukashin, C.↗

NASA NTRS2018

Improving NASA Earth Science Data and Information Access Through Natural Language Processing Based Data Analysis and Visualization

NASA: The Research Access initiative is part of the agency's framework for increasing public access to scientific publications and digital scientific data. The initiative follows the release of White House Office of Science and Technology Policy's (OSTP) memorandum "Increasing Access to the Results of Federally Funded Research," to ensure federally funded research is available to the public within one year of publication. NASA answered the mandate by creating an agency plan entitled "NASA Plan for Increasing Access to the Results of Scientific Research" and associated policy, NPD 2230.1, Research Data and Publication Access. Principles in NASA SMD Strategic Plan for Scientific Data and Computing: Continued free and open access to scientific data for any use. Improved ease of use and discoverability. Enhanced science applications and new use cases. Incorporates best practices and "state of the art" through partnerships. Earth Data and Systems are Evolving: Increasing archive and file sizes. More complicated data structures. More user-friendly and data services. What is the future direction?

Liu, Zhong↗

DOE OSTI2021

Hyper Suprime-Cam Legacy Archive

Abstract We present the launch of the Hyper Suprime-Cam Legacy Archive (HSCLA), a public archive of processed, science-ready data from the Hyper Suprime-Cam (HSC). HSC is an optical wide-field imager installed at the prime focus of the Subaru Telescope and has been in operation since 2014. While ∼1/3 of the total observing time of HSC has been used for the Subaru Strategic Program (SSP), the remainder of the time is used for Principal Investigator (PI)-based programs. We have processed the data from these PI-based programs and make the processed, high-quality data available to the community through HSCLA. The current version of HSCLA includes data taken in the first year of science operation, 2014. We provide both individual and coadd images as well as photometric catalogs. The photometric catalog from the coadd is loaded to the database, which offers a fast access to the large catalog. There are other online tools such as an image browser and an image cutout tool and they will be useful for science analyses. The coadd images reach 24–27th magnitudes at 5σ for point sources and cover approximately 580 square degrees in at least one filter with 150 million objects in total. We perform extensive quality assurance tests and verify that the photometric and astrometric quality of the data is good enough for most scientific explorations. However, the data are not without problems and users are referred to the list of known issues before exploiting the data for science. All the data and documentations can be found at the data release site, 〈https://hscla.mtk.nao.ac.jp/〉.

Tanaka, Masayuki↗

NASA NTRS2016

Earth Science Data Analytics: Preparing for Extracting Knowledge from Information

Data analytics is the process of examining large amounts of data of a variety of types to uncover hidden patterns, unknown correlations and other useful information. Data analytics is a broad term that includes data analysis, as well as an understanding of the cognitive processes an analyst uses to understand problems and explore data in meaningful ways. Analytics also include data extraction, transformation, and reduction, utilizing specific tools, techniques, and methods. Turning to data science, definitions of data science sound very similar to those of data analytics (which leads to a lot of the confusion between the two). But the skills needed for both, co-analyzing large amounts of heterogeneous data, understanding and utilizing relevant tools and techniques, and subject matter expertise, although similar, serve different purposes. Data Analytics takes on a practitioners approach to applying expertise and skills to solve issues and gain subject knowledge. Data Science, is more theoretical (research in itself) in nature, providing strategic actionable insights and new innovative methodologies. Earth Science Data Analytics (ESDA) is the process of examining, preparing, reducing, and analyzing large amounts of spatial (multi-dimensional), temporal, or spectral data using a variety of data types to uncover patterns, correlations and other information, to better understand our Earth. The large variety of datasets (temporal spatial differences, data types, formats, etc.) invite the need for data analytics skills that understand the science domain, and data preparation, reduction, and analysis techniques, from a practitioners point of view. The application of these skills to ESDA is the focus of this presentation. The Earth Science Information Partners (ESIP) Federation Earth Science Data Analytics (ESDA) Cluster was created in recognition of the practical need to facilitate the co-analysis of large amounts of data and information for Earth science. Thus, from a to advance science point of view: On the continuum of ever evolving data management systems, we need to understand and develop ways that allow for the variety of data relationships to be examined, and information to be manipulated, such that knowledge can be enhanced, to facilitate science. Recognizing the importance and potential impacts of the unlimited ways to co-analyze heterogeneous datasets, now and especially in the future, one of the objectives of the ESDA cluster is to facilitate the preparation of individuals to understand and apply needed skills to Earth science data analytics. Pinpointing and communicating the needed skills and expertise is new, and not easy. Information technology is just beginning to provide the tools for advancing the analysis of heterogeneous datasets in a big way, thus, providing opportunity to discover unobvious scientific relationships, previously invisible to the science eye. And it is not easy It takes individuals, or teams of individuals, with just the right combination of skills to understand the data and develop the methods to glean knowledge out of data and information. In addition, whereas definitions of data science and big data are (more or less) available (summarized in Reference 5), Earth science data analytics is virtually ignored in the literature, (barring a few excellent sources).

data analytics↗

NASA NTRS2007

A Contrast in Use of Metrics in Earth Science Data Systems

In recent years there has been a surge in the number of systems for processing, archiving and distributing remotely sensed data. Such systems, working independently as well as in collaboration, have been contributing greatly to the advances in the scientific understanding of the Earth system, as well as utilization of the data for nationally and internationally important applications. Among such systems, we consider those that are developed by or under the sponsorship of NASA to fulfill one of its strategic objectives: "Study Earth from space to advance scientific understanding and meet societal needs." NASA's Earth science data systems are of varying size and complexity depending on the requirements they are intended to meet. Some data systems are regarded as NASA's "Core Capabilities" that provide the basic infrastructure for processing, archiving and distributing a set of data products to a large and diverse user community in a robust and reliable manner. Other data systems constitute "Community Capabilities". These provide specialized and innovative services to data users and/or research products offering new scientific insight. Such data systems are generally supported by NASA through peer reviewed competition. Examples of Core Capabilities are 1. Earth Observing Data and Information System (EOSDIS) with its Distributed Active Archive Centers (DAACs), Science Investigator-led Processing Systems (SIPSs), and the EOS Clearing House (ECHO); 2. Tropical Rainfall Measurement Mission (TRMM) Science Data and Information System (TSDIS); 3. Ocean Data Processing System (ODPS); and 4. CloudSat Data Processing Center. Examples of Community Capabilities are projects under the Research, Education and Applications Solutions Network (REASON), and Advancing Collaborative Connections for Earth System Science (ACCESS) Programs. In managing these data system capabilities, it is necessary to have well-established goals and to measure progress relative to them. Progress is measured through "metrics", which can be a combination of quantitative as well as qualitative assessments. The specific metrics of interest depend on the user of the metrics as well as the type of data system. The users of metrics can be data system managers, program managers, funding agency or the public. Data system managers need metrics for assessing and improving the performance of the system and for future planning. Program managers need metrics to assess progress and the value of the data systems sponsored by them. Also, there is a difference in the metrics needed for core capabilities that tend to be more complex, larger and longer-term compared to community capabilities and the community capabilities that tend to be simpler, smaller and shorter-term. Even among community capabilities there are differences; hence the same set of metrics does not apply to all. Some provide data products to users, some provide services that enable better utilization of data or interoperability among other systems, and some are a part of a larger project where provision of data or services is only a minor activity. There is also a contrast between metrics used for internal and external purposes. Examples of internal purposes are: ensuring that the system meets its requirements, and planning for evolution and growth. Examples of external purposes are: providing to sponsors indicators of success of the systems, demonstrating the contributions of the system to overall program success, etc. This paper will consider EOSDIS, REASON and ACCESS programs to show the various types of metrics needed and how they need to be tailored to the types of data systems while maintaining the overall management goals of measuring progress and contributions made by the data systems.

Ramapriyan, Hampapuram↗

DOE OSTI2022

An Overview in the Development of a Multi-Sensor Data Science System for Monitoring a Solvent Extraction Process

To enhance nuclear nonproliferation stewardship, Idaho National Laboratory is building the Beartooth nuclear fuel cycle processing test bed. The Beartooth test bed will allow researchers the opportunity to study nuclear fuel processing operations including the use of centrifugal contactors in solvent extraction processes. The test bed is designed to support data collection and machine learning to monitor process operations. As part of this project, researchers will study data collected from a host of sensors that have not been typically used to monitor solvent extraction processes such as vibration, acoustic, colormetric, and thermal measurement data. The goal of this research is to employ machine learning and data analytics to study the confluence of signals collected from both traditionally and non-traditionally used sensors to provide operator process awareness and discover process anomalies. An overview of planned sensors and experiments that will focus on signal discovery will be presented.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2022

An Overview in the Development of a Multi-Sensor Data Science System for Monitoring a Solvent Extraction Process

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2023

Development of a Multi Sensor Data Science System for Monitoring a Solvent Extraction Process

Idaho National Laboratory is designing and constructing a solvent extraction test bed, Beartooth, as part of a nuclear fuel cycle stewardship initiative. The test bed will provide facilities and equipment for testing novel extraction processes and give early career scientists opportunities to gain skills in performing separations chemistry. The test bed is being uniquely designed to enable machine learning capabilities for the characterization of chemical process operations. One of the intentions of incorporating machine learning methods is to provide process operators with enhanced situational awareness for the optimization of separations techniques and possible detection of a diversion event. As a result, the objective of this Laboratory Directed Research and Development project is to develop a multi-sensor data collections system and implement machine learning techniques on signals from an existing centrifugal contactor cascade to identify equipment usage and process events. This presentation will provide an overview of sensors used and experiments conducted to date.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

NASA NTRS1996

Technology Directions for the 21st Century

The Office of Space Communications (OSC) is tasked by NASA to conduct a planning process to meet NASA's science mission and other communications and data processing requirements. A set of technology trend studies was undertaken by Science Applications International Corporation (SAIC) for OSC to identify quantitative data that can be used to predict performance of electronic equipment in the future to assist in the planning process. Only commercially available, off-the-shelf technology was included. For each technology area considered, the current state of the technology is discussed, future applications that could benefit from use of the technology are identified, and likely future developments of the technology are described. The impact of each technology area on NASA operations is presented together with a discussion of the feasibility and risk associated with its development. An approximate timeline is given for the next 15 to 25 years to indicate the anticipated evolution of capabilities within each of the technology areas considered. This volume contains four chapters: one each on technology trends for database systems, computer software, neural and fuzzy systems, and artificial intelligence. The principal study results are summarized at the beginning of each chapter.

Crimi, Giles F.↗

DOE OSTI2023

Preliminary Results of a Multi-Sensor Data Science System for Monitoring a Solvent Extraction Process

Idaho National Laboratory is building a test bed to allow researchers the opportunity to study nuclear fuel processing operations. This includes studying the solvent extraction process and the use of centrifugal contactors. The goal of this project is to develop a system that utilizes non-traditional measurement sources such as vibration, acoustics, current, color, flow, and temperature in conjunction with data-based, machine learning techniques that will allow for signal discovery. This multi-sensor data supports the development of safeguards by design, provides operator process awareness, and aids in the discovery of process anomalies. This paper highlights some of the preliminary results from initial data collection campaigns and shares some of the lessons learned.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗