Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “science data processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,153 records · Page 64
All sourcesNASA NTRSDOE OSTI
NASA NTRS2014

Multilateral Biomedical Data Sharing in the One-year Joint US-Russian Mission on the International Space Station

The One Year Mission (1YM) by two astronauts on the International Space Station (ISS), starting in March 2015, offers a unique opportunity to expand multilateral collaboration by sharing data and resources among the partner agencies in preparation for planned space exploration missions beyond low Earth orbit. Agreements and protocols will be established for the collection, distribution, analysis and reporting of both research and clinical data. Data will be shared between the agencies sponsoring the investigators, and between the research and clinical medicine communities where common interests are identified. The assignment of only two astronauts, one Russian and the other American, to the 1YM necessitated creativity in bilateral efforts to maximize the biomedical return from the opportunity. Addition of Canadian, European and Japanese investigations make the effort even more integrative. There will be three types of investigations: joint, cross‐participation and data‐exchange. The joint investigations have US and Russian coprincipal investigators, and the data acquired will be their common responsibility. The other two types must develop data sharing agreements and processes specific to their needs. A multilateral panel of ISS partner space agencies will develop policies for international exchange of scientific information to meet their science objectives and priorities. They will promote archiving of space flight data and will inform each other and the scientific community at large about the results obtained from space life sciences studies. Integration tasks for the 1YM are based on current experience from the ISS and previous efforts on the Russian space station Mir. Closer coordination between international partners requires more common approaches to remove barriers to multilateral resource utilization on the ISS. Greater integration in implementation should increase utilization efficiency to benefit all participants in spaceflight human research. This presentation will describe the overarching principles for multilateral data collection, analysis and sharing and for data security for medical and research data shared between ISS partners prior to release in public forums.

Charles, John B.↗
NASA NTRS2004

The Montage architecture for grid-enabled science processing of large, distributed datasets

Montage is an Earth Science Technology Office (ESTO) Computational Technologies (CT) Round III Grand Challenge investigation to deploy a portable, compute-intensive, custom astronomical image mosaicking service for the National Virtual Observatory (NVO). Although Montage is developing a compute- and data-intensive service for the astronomy community, we are also helping to address a problem that spans both Earth and Space science, namely how to efficiently access and process multi-terabyte, distributed datasets. In both communities, the datasets are massive, and are stored in distributed archives that are, in most cases, remote from the available Computational resources. Therefore, state of the art computational grid technologies are a key element of the Montage portal architecture. This paper describes the aspects of the Montage design that are applicable to both the Earth and Space science communities.

virtual observatory↗
DOE OSTI2020

Software-defined network for end-to-end networked science at the exascale

Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. The work presented here is motivated by a vision for a new smart network and smart application ecosystem that will provide a more deterministic and interactive environment for domain science workflows. The Software-Defined Network for End-to-end Networked Science at Exascale (SENSE) system includes a model-based architecture, implementation, and deployment which enables automated end- to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements, an intelligent orchestrator and resource control systems allow for custom tailoring of scalability and real-time responsiveness based on individual application and infrastructure operator requirements. This allows the science applications to manage the network as a first-class schedulable resource as is the current practice for instruments, compute, and storage systems. Deployment and experiments on production networks and testbeds have validated SENSE functions and performance. Emulation based testing verified the scalability needed to support research and education infrastructures. Key contributions of this work include an architecture definition, reference implementation, and deployment. This provides the basis for further innovation of smart network services to accelerate scientific discovery in the era of big data, cloud computing, machine learning and artificial intelligence.

47 OTHER INSTRUMENTATION↗
NASA NTRS1992

Remote Earth Sciences data collection using ACTS

Given the focus on global change and the attendant scope of such research, we anticipate significant growth of requirements for investigator interaction, processing system capabilities, and availability of data sets. The increased complexity of global processes requires interdisciplinary teams to address them; the investigators will need to interact on a regular basis; however, it is unlikely that a single institution will house sufficient investigators with the required breadth of skills. The complexity of the computations may also require resources beyond those located within a single institution; this lack of sufficient computational resources leads to a distributed system located at geographically dispersed institutions. Finally the combination of long term data sets like the Pathfinder datasets and the data to be gathered by new generations of satellites such as SeaWiFS and MODIS-N yield extra-ordinarily large amounts of data. All of these factors combine to increase demands on the communications facilities available; the demands are generating requirements for highly flexible, high capacity networks. We have been examining the applicability of the Advanced Communications Technology Satellite (ACTS) to address the scientific, computational, and, primarily, communications questions resulting from global change research. As part of this effort three scenarios for oceanographic use of ACTS have been developed; a full discussion of this is contained in Appendix B.

Evans, Robert H.↗
DOE OSTI2020

Software-Defined Network for End-to-end Networked Science at the Exascale

Domain science applications and workflow processes are currently forced to view the network as an opaque infrastructure into which they inject data and hope that it emerges at the destination with an acceptable Quality of Experience. There is little ability for applications to interact with the network to exchange information, negotiate performance parameters, discover expected performance metrics, or receive status/troubleshooting information in real time. The work we presen here is motivated by a vision for a new smart network and smart application ecosystem that will provide a more deterministic and interactive environment for domain science workflows. The Software-Defined Network for End-to-end Networked Science at Exascale (SENSE) system includes a model-based architecture, implementation, and deployment which enables automated end-to-end network service instantiation across administrative domains. An intent based interface allows applications to express their high-level service requirements, an intelligent orchestrator and resource control systems allow for custom tailoring of scalability and real-time responsiveness based on individual application and infrastructure operator requirements. This allows the science applications to manage the network as a first-class schedulable resource as is the current practice for instruments, compute, and storage systems. Deployment and experiments on production networks and testbeds have validated SENSE functions and performance. Emulation based testing verified the scalability needed to support research and education infrastructures. Key contributions of this work include an architecture definition, reference implementation, and deployment. This provides the basis for further innovation of smart network services to accelerate scientific discovery in the era of big data, cloud computing, machine learning and artificial intelligence.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Leveraging Environmental Research and Observation Networks to Advance Soil Carbon Science

Soil organic matter (SOM) is a critical ecosystem variable regulated by interacting physical, chemical, and biological processes. Collaborative efforts to integrate perspectives, data, and models from interdisciplinary research and observation networks can significantly advance predictive understanding of SOM. We outline how integrating three networks—the Long-Term Ecological Research with a focus on ecological dynamics, the Critical Zone Observatories with strengths in landscape/geologic context, and the National Ecological Observatory Network with standardized multiscale measurements—can advance SOM knowledge. This integration requires improved data dissemination and sharing, coordinated data collection activities, and enhanced collaboration between empiricists and modelers within and across networks.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2017

A Collimated CZT Detector for Quantitative Gamma Assays - 17250

Cadmium-Zinc-Telluride [CZT] detectors for gamma spectroscopic measurements and room-temperature operation have had a long history of promising good performance. Very small detectors [0.05-0.5 cm{sup 3}] have been commercially available for many years, but their small size creates problems for spectroscopy because of very low photopeak efficiency, especially at medium and high energies. Large CZT [4-6 cm{sup 3}] detectors have been shown to work nicely in a research environment when complicated electronic and data processing algorithms are applied. We report here test results from a commercially available 1 cubic centimeter CZT detector housed in a portable custom Tungsten shield with a set of collimators with different apertures. The detector has 2-2.5% FWHM resolution at Cs-137 energies, and near-Gaussian shaped peaks. Efficiency calibrations can be quickly made for a wide variety of geometries using the ISOCS mathematical efficiency software. We show successful validation tests comparing accuracy of measurements on reference calibration point sources, line sources, and 200 liter drums. The low weight of the device [9 kg including the shield and collimator] and low power [powered via USB from the PC] makes these devices particularly suitable for human-deployed in-situ measurements of large objects containing medium levels of activity - e.g. characterization of objects encountered during decommissioning, emergency response, truck alarm investigation, or waste being shipped to a disposal facility. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS2001

The CCSDS Lossless Data Compression Algorithm for Space Applications

In the late 80's, when the author started working at the Goddard Space Flight Center (GSFC) for the National Aeronautics and Space Administration (NASA), several scientists there were in the process of formulating the next generation of Earth viewing science instruments, the Moderate Resolution Imaging Spectroradiometer (MODIS). The instrument would have over thirty spectral bands and would transmit enormous data through the communications channel. This was when the author was assigned the task of investigating lossless compression algorithms for space implementation to compress science data in order to reduce the requirement on bandwidth and storage.

Yeh, Pen-Shu↗
NASA NTRS1992

AVIRIS ground data processing system

During the last year and a half, Feb. 1991 to Jun. 1992, a major upgrade of the Airborne Visible/Infrared Imaging Spectrometers (AVIRIS) ground data processing system took place. Both the hardware and software components were changed significantly to improve the processing capacity and performance and to structure a data facility capable of handling the projected work load into the near future. A summary report of these changes and some projections for the future are provided. The objectives of the AVIRIS data facility are to decommutate and archive AVIRIS data and to provide raw or radiometrically calibrated data products to the science investigator. These primary objectives have not changed from the initial concepts. The upgrade effort has greatly improved the processing system. These objectives can now be accomplished in a more timely fashion at a reasonable cost and there is sufficient capacity to manage the current processing load and provide for future growth. The method of implementation added the flexibility to provide better service to the investigator and allow for future changes.

Hansen, Earl G.↗
NASA NTRS2004

Spaceflight Operations Services Grid (SOSG) Prototype Implementation and Feasibility Study

Science Operations Services Grid is focusing on building a prototype grid-based environment that incorporates existing and new spaceflight services to enable current and future NASA programs with cost savings and new and evolvable methods to conduct science in a distributed environment. The Science Operations Services Grid (SOSG) will provide a distributed environment for widely disparate organizations to conduct their systems and processes in a more efficient and cost effective manner. These organizations include those that: 1) engage in space-based science and operations, 2) develop space-based systems and processes, and 3) conduct scientific research, bringing together disparate scientific disciplines like geology and oceanography to create new information. In addition educational outreach will be significantly enhanced by providing to schools the same tools used by NASA with the ability of the schools to actively participate on many levels in the science generated by NASA from space and on the ground. The services range from voice, video and telemetry processing and display to data mining, high level processing and visualization tools all accessible from a single portal. In this environment, users would not require high end systems or processes at their home locations to use these services. Also, the user would need to know minimal details about the applications in order to utilize the services. In addition, security at all levels is an underlying goal of the project. The Science Operations Services Grid will focus on four tools that are currently used by the ISS Payload community along with nine more that are new to the community. Under the prototype four Grid virtual organizations PO) will be developed to represent four types of users. They are a Payload (experimenters) VO, a Flight Controllers VO, an Engineering and Science Collaborators VO and an Education and Public Outreach VO. The User-based services will be implemented to replicate the operational voice, video, telemetry and commanding systems. Once the User-based services are in place, they will be analyzed to establish feasibility for Grid enabling. If feasible then each User-based service will be Grid enabled. The remaining non-Grid services if not already Web enabled will be so enabled. In the end, four portals will be developed one for each VO. Each portal will contain the appropriate User-based services required for that VO to operate.

Bradford, Robert N.↗
DOE OSTI2016

Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

The.LHC, operating at CERN, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. While PanDA currently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, LHC data taking runs require more resources than grid can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, in particular with Titan supercomputer at Oak Ridge Leadership Computing Facility. Current approach utilizes modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single threaded workloads in parallel on LCFs multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms for ALICE and ATLAS experiments and it is in full pro duction for the ATLAS since September 2015. We will present our current accomplishments with running PanDA at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

INTEGRATION OF PANDA WORKLOAD MANAGEMENT SYSTEM WITH SUPERCOMPUTERS

Abstract The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the funda- mental nature of matter and the basic forces that shape our universe, and were recently credited for the dis- covery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 140 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data cen- ters are physically scattered all over the world. While PanDA currently uses more than 250000 cores with a peak performance of 0.3+ petaFLOPS, next LHC data taking runs will require more resources than Grid computing can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, Europe and Russia (in particular with Titan supercomputer at Oak Ridge Leadership Com- puting Facility (OLCF), Supercomputer at the National Research Center Kurchatov Institute , IT4 in Ostrava, and others). The current approach utilizes a modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single- threaded workloads in parallel on Titan s multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms. We will present our current accom- plishments in running PanDA WMS at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facility s infrastructure for High Energy and Nuclear Physics, as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

De, K↗
DOE OSTI2019

Event Detection for Undersampled Electron Microscopy Experiments: A Control Chart Case Study

Real-time systems often consist of high-volume, high-velocity data streams in which making efficient and timely inferences about the data are challenging. In particular, when a user interacts with a real-time system to gain insights, detect events, and make decisions about the system, the rate and amount of information the user is required to process is generally overwhelming. In addition, analytically processing large volumes of data can be computationally expensive and, in real-time, renders traditional inferential methods effectively useless. One approach to mitigate these challenges is to reduce both the amount of information presented to the user and the volume of data placed in the stream. In this manuscript we will describe a method constructed under this approach in addition to the development and deployment of an online tool, Real time Event Detector for Subsampled Images (REDSI), designed to provide feedback on a real-time system by characterizing and detecting events of interest. We will discuss REDSI in the context of scanning transmission electron microscopy (STEM), which is a powerful real-time system that provides high spatial and temporal resolution on nanoscale structures and processes. The data produced by STEM experiments are a stream of images relaying structural, compositional, and dynamic interphase information to scientists in fields ranging from microbiology and neuroscience to materials science and energetics. While REDSI was designed to processes STEM images in real time, we will also discuss other real-time system applications for this methodology.

multivariate statistics, event detection↗
DOE OSTI2026

Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors (Phase-I)

With an ever increasing demand for high precision data from modern detectors for discovery science and precision measurements, all major high energy nuclear and particle experiments, current and future, are facing the challenge on how to deal with the large volume of raw data generated from sophisticated state-of-the-art detectors in high rate collisions. These goals need to be balanced with available hardware and cost limits on DAQ (Data AcQuisition system) bandwidth and offline computing resources to capture, store and process the signal events. Two prototypical examples are the upcoming sPHENIX experiment, the DOE next generation heavy ion physics experiment at the Relativistic Heavy Ion Collider at BNL, and the future EIC experiments that are planned to be online circa 2030.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2016

Expanding the role of reactive transport models in critical zone processes

We report that models test our understanding of processes and can reach beyond the spatial and temporal scales of measurements. Multi-component Reactive Transport Models (RTMs), initially developed more than three decades ago, have been used extensively to explore the interactions of geothermal, hydrologic, geochemical, and geobiological processes in subsurface systems. Driven by extensive data sets now available from intensive measurement efforts, there is a pressing need to couple RTMs with other community models to explore non-linear interactions among the atmosphere, hydrosphere, biosphere, and geosphere. Here we briefly review the history of RTM development, summarize the current state of RTM approaches, and identify new research directions, opportunities, and infrastructure needs to broaden the use of RTMs. In particular, we envision the expanded use of RTMs in advancing process understanding in the Critical Zone, the veneer of the Earth that extends from the top of vegetation to the bottom of groundwater. We argue that, although parsimonious models are essential at larger scales, process-based models offer tools to explore the highly nonlinear coupling that characterizes natural systems. We present seven testable hypotheses that emphasize the unique capabilities of process-based RTMs for (1) elucidating chemical weathering and its physical and biogeochemical drivers; (2) understanding the interactions among roots, micro-organisms, carbon, water, and minerals in the rhizosphere; (3) assessing the effects of heterogeneity across spatial and temporal scales; and (4) integrating the vast quantity of novel data, including “omics” data (genomics, transcriptomics, proteomics, metabolomics), elemental concentration and speciation data, and isotope data into our understanding of complex earth surface systems. In conclusion, with strong support from data-driven sciences, we are now in an exciting era where integration of RTM framework into other community models will facilitate process understanding across disciplines and across scales.

54 ENVIRONMENTAL SCIENCES↗