Echo: Data and Analysis Management
This document describes, Echo, a comprehensive suite of tools for data wrangling, management, and analysis, and its application.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
This document describes, Echo, a comprehensive suite of tools for data wrangling, management, and analysis, and its application.
The United States 2020 Space Policy directive declares that NASA, in cooperation with private industry, will “extend human economic activity into deep space by establishing a permanent human presence on the Moon”. This goal will require advanced data management, as well as analysis, modeling and representation of lunar information in order to prepare for Artemis human missions, lunar science investigations and exploration. To meet this requirement, we conceptualize and present an implementation strategy for a distributed platform for lunar data retrieval, inferencing and analysis, which will be based on federated learning and the NASA Celestial Mapping System (CMS). In addition to demonstrating the imperative of enabling lunar-borne data to remain in-situ but still accessible, this presentation will also include examples of how third parties could contribute both datasets and new functionality into this platform using an AI-based data import pipeline and a plug-in architecture respectively.
The OECD/NEA Radioactive Waste Repository Metadata Management (RepMet) initiative aims to bring about a better understanding of the identification and administration of metadata - a key aspect of data management - to support national programmes in managing their radioactive waste repository data, information and records in a way that is both harmonised internationally and suitable for long-term management and use. At this point in the project, the main focus lies on 'waste packages at their storage sites ready for geological disposal' but excluding spent fuel and high level waste (HLW). Later work will deal with metadata collected on spent fuel and HLW as well as with different phases of repository development: design and construction, and operation. In order to develop metadata, RepMet members have provided lists of data items they currently collect and manage for packaged waste. The lists, albeit it non comprehensive, suggest that the number and type of data items varies importantly from organisation to organisation. At term, RepMet plans to generate a comprehensive list of such data items and their metadata. In parallel, RepMet is developing a document which examines a selected number of existing metadata standards. Because of the specialist nature of radioactive waste management, including waste processing facilities, disposal facilities and waste packages, existing metadata standards do not seem to be transferable as defined. Rather, RepMet will explore such standards to identify relevant principles for possible adaptation to the radioactive waste management community. (authors)
This report presents the data qualification status of fuel irradiation data from the first four reactor cycles (147A, 148A, 148B, and 149A) of the on-going second Advanced Gas Reactor (AGR-2) experiment as recorded in the NGNP Data Management and Analysis System (NDMAS). This includes data received by NDMAS from the period June 22, 2010 through May 21, 2011. AGR-2 is the second in a series of eight planned irradiation experiments for the AGR Fuel Development and Qualification Program, which supports development of the very high temperature gas-cooled reactor (VHTR) under the Next Generation Nuclear Plant (NGNP) Project. Irradiation of the AGR-2 test train is being performed at the Advanced Test Reactor (ATR) at the Idaho National Laboratory (INL) and is planned for 600 effective full power days (approximately 2.75 calendar years) (PLN-3798). The experiment is intended to demonstrate the performance of UCO (uranium oxycarbide) and UO2 (uranium dioxide) fuel produced in a large coater. Data qualification status of the AGR-1 experiment was reported in INL/EXT-10-17943 (Abbott et al. 2010).
Data-driven assessments and adaptive feedback are becoming a cornerstone research in educational data analytics and involve developing methods for exploring the unique types of data that come from the educational context. For example, predicting college student performance is crucial for both the students and educational institutions. It can support timely intervention to prevent students from failing a course, increasing efficacy of advising functions, and improving course completion rate. In this paper, we present our efforts in using data analytics that enable educationists to design novel data-driven assessment and feedback mechanisms. In order to achieve this objective, we investigate temporal stability of students grades and perform predictive analytics on academic data collected from 2009 through 2013 in one of the most commonly used learning management systems, called Moodle. First, we have identified the data features useful for assessments and predicting student outcomes such as students scores in homework assignments, quizzes, exams, in addition to their activities in discussion forums and their total Grade Point Average(GPA) at the same term they enrolled in the course. Second, time series models in both frequency and time domains are applied to characterize the progression as well as overall projections of the grades. In particular, the model analyzed the stability as well as fluctuation of grades among students during the collegiate years (from freshman to senior) and disciplines. Third, Logistic Regression and Neural Network predictive models are used to identify students as early as possible who are in danger of failing the course they are currently enrolled in. These models compute the likelihood of any given student failing (or passing) the current course. The time series analysis indicates that assessments and continuous feedback are critical for freshman and sophomores (even with easy courses) than for seniors, and those assessments may be provided using the predictive models. Numerical results are presented to evaluate and compare the performance of the developed models and their predictive accuracy. Our results show that there are strong ties associated with the first few weeks for coursework and they have an impact on the design and distribution of individual modules.
A foundational set of findable, accessible, interoperable, and reusable (FAIR) principles were proposed in 2016 as prerequisites for proper data management and stewardship, with the goal of enabling the reusability of scholarly data. The principles were also meant to apply to other digital assets, at a high level, and over time, the FAIR guiding principles have been re-interpreted or extended to include the software, tools, algorithms, and workflows that produce data. FAIR principles are now being adapted in the context of AI models and datasets. Here, we present the perspectives, vision, and experiences of researchers from different countries, disciplines, and backgrounds who are leading the definition and adoption of FAIR principles in their communities of practice, and discuss outcomes that may result from pursuing and incentivizing FAIR AI research. The material for this report builds on the FAIR for AI Workshop held at Argonne National Laboratory on June 7, 2022.
Global gridded crop models (GGCMs) combine agronomic or plant growth models with gridded spatial input data to estimate spatially explicit crop yields and agricultural externalities at the global scale. Differences in GGCM outputs arise from the use of different biophysical models, setups, and input data. GGCM ensembles are frequently employed to bracket uncertainties in impact studies without investigating the causes of divergence in outputs. This study explores differences in maize yield estimates from five GGCMs based on the public domain field-scale model Environmental Policy Integrated Climate (EPIC) that participate in the AgMIP Global Gridded Crop Model Intercomparison initiative. Albeit using the same crop model, the GGCMs differ in model version, input data, management assumptions, parameterization, and selection of subroutines affecting crop yield estimates via cultivar distributions, soil attributes, and hydrology among others. The analyses reveal inter-annual yield variability and absolute yield levels in the EPIC-based GGCMs to be highly sensitive to soil parameterization and crop management. All GGCMs show an intermediate performance in reproducing reported yields with a higher skill if a static soil profile is assumed or sufficient plant nutrients are supplied. An in-depth comparison of setup domains for two EPIC-based GGCMs shows that GGCM performance and plant stress responses depend substantially on soil parameters and soil process parameterization, i.e. hydrology and nutrient turnover, indicating that these often neglected domains deserve more scrutiny. For agricultural impact assessments, employing a GGCM ensemble with its widely varying assumptions in setups appears the best solution for coping with uncertainties from lack of comprehensive global data on crop management, cultivar distributions and coefficients for agro-environmental processes. However, the underlying assumptions require systematic specifications to cover representative agricultural systems and environmental conditions. Furthermore, the interlinkage of parameter sensitivity from various domains such as soil parameters, nutrient turnover coefficients, and cultivar specifications highlights that global sensitivity analyses and calibration need to be performed in an integrated manner to avoid bias resulting from disregarded core model domains. Finally, relating evaluations of the EPIC-based GGCMs to a wider ensemble based on individual core models shows that structural differences outweigh in general differences in configurations of GGCMs based on the same model, and that the ensemble mean gains higher skill from the inclusion of structurally different GGCMs. Although the members of the wider ensemble herein do not consider crop-soil-management interactions, their sensitivity to nutrient supply indicates that findings for the EPIC-based sub-ensemble will likely become relevant for other GGCMs with the progressing inclusion of such processes.
SynTrack is a dynamic, workflow-driven data management system that tracks the DNA build process: Management of the hierarchical relationships of the DNA fragments; Monitoring of process tasks for the assembly of multiple DNA fragments into final constructs; Creations of vendor order forms with selectable building blocks. Organizing plate layouts barcodes for vendor/pcr/fusion/chewback/bioassay/glycerol/master plate maps (default/condensed); Creating or updating Pre-Assembly/Assembly process workflows with selected building blocks; Generating Echo pooling instructions based on plate maps; Tracking of building block orders, received and final assembled for delivering; Bulk updating of colony or PCR amplification information, fusion PCR and chewback results; Updating with QA/QC outcome with .csv & .xlsx template files; Re-work assembly workflow enabled before and after sequencing validation; and Tracking of plate/well data changes and status updates and reporting of master plate status with QC outcomes.
Effective implementation of technology that facilitates accurate and high-throughput screening of thousands of field-grown lines is critical for accelerating crop improvement and breeding strategies for higher yield and disease tolerance. Progress in the development of field-based high throughput phenotyping methods has advanced considerably in the last 10 years through technological progress in sensor development and high-performance computing. In this paper, we review recent advances in high throughput field phenotyping technologies designed to inform the genetics of quantitative traits, including crop yield and disease tolerance. Successful application of phenotyping platforms to advance crop breeding and identify and monitor disease requires: (1) high resolution of imaging and environmental sensors; (2) quality data products that facilitate computer vision, machine learning and GIS; (3) capacity infrastructure for data management and analysis; and (4) automated environmental data collection. Finally, accelerated breeding for agriculturally relevant crop traits is key to the development of improved varieties and is critically dependent on high-resolution, high-throughput field-scale phenotyping technologies that can efficiently discriminate better performing lines within a larger population and across multiple environments.
Mercury is a federated metadata harvesting, search and retrieval tool based on both open source software and software developed at Oak Ridge National Laboratory. It was originally developed for NASA, and the Mercury development consortium now includes funding from NASA, USGS, and DOE. A major new version of Mercury (version 3.0) was developed during 2007 and released in early 2008. This Mercury 3.0 version provides orders of magnitude improvements in search speed, support for additional metadata formats, integration with Google Maps for spatial queries, facetted type search, support for RSS delivery of search results, and ready customization to meet the needs of the multiple projects which use Mercury. For the end users, Mercury provides a single portal to very quickly search for data and information contained in disparate data management systems. It collects metadata and key data from contributing project servers distributed around the world and builds a centralized index. The Mercury search interfaces then allow the users to perform simple, fielded, spatial, and temporal searches across these metadata sources. This centralized repository of metadata with distributed data sources provides extremely fast search results to the user, while allowing data providers to advertise the availability of their data and maintain complete control and ownership of that data.
Numerical representation of the geologic framework and its hydrologic and geochemical properties is an integral part of all vadose zone flow and transport modeling. Historically, the geologic framework and the physical distribution of flow and transport properties have been represented by simple homogeneous and horizontally stratified hydrogeologic units. To capture more of the heterogeneity, small-scale variability, and uncertainty within a model, the physical and geochemical parameters may be represented by probability density functions loosely correlated to individual hydrogeologic units. As computer-processing capabilities have become more advanced, there has been more emphasis on improving spatial resolution and quantifying uncertainty in key model parameters. One of the more popular approaches has focused on geostatistical simulation of the flow and transport properties themselves, with little regard to the geologic strata and sedimentary sequences. However, newer approaches are focusing more on geostatistical simulation of the sequence-stratigraphic relations of lithofacies and the geostatistical distributions of flow and transport properties within those facies. These approaches require more rigorous quantitative treatment of geologic data than are normally supported by the mostly qualitative nature of borehole geologic information. Thus at the Hanford Site, efforts are being made to systematize borehole geologic data to be used in a more quantitative way to define the spatial distribution of flow and transport properties in support of vadose flow and transport simulations. New detailed procedures translate qualitative descriptive information into categorical data, and translate inconsistent quantitative and semi-quantitative data into common parametric data sets. A geologic data management system is being developed to manage and integrate these new interpreted categorical data sets with other existing databases to support synergistic analysis and to improve numerical representation of the geologic architecture.
The Graphite Technology Development Program will run a series of six experiments to quantify the effects of irradiation on nuclear grade graphite. The third experiment, Advanced Graphite Creep 3 (AGC 3), began with Advanced Test Reactor (ATR) Cycle 152B on November 27, 2012, and ended with ATR Cycle 155B on April 23, 2014. This report documents qualification of AGC 3 experiment irradiation monitoring data for use by the Very High Temperature Reactor (VHTR) Technology Development Office (TDO) Program for research and development activities required to design and license the first VHTR nuclear plant. Qualified data meet the requirements for data collection and use as described in the experiment planning and quality assurance documents. Failed data do not meet the requirements. Trend data may not meet the requirements, but may still provide some useable information. All thermocouples (TCs) functioned throughout the AGC 3 experiment. There was one interval between December 18, 2012, and December 20, 2012, where 10 NULL values were reported for various TCs. These NULL values were deleted from the Nuclear Data Management and Analysis System database. All temperature data are Qualified for use by the VHTR TDO Program. Argon, helium, and total gas flow data were within expected ranges and are Qualified for use by the VHTR TDO Program. Total gas flow was approximately 50 sccm through the AGC 3 experiment capsule. Helium gas flow was briefly increased to 100 sccm during ATR shutdowns. At the start of the AGC 3 experiment, moisture in the outflow gas line was stuck at a constant value of 335.6174 ppmv for the first cycle (Cycle 152B). When the AGC 3 experiment capsule was reinstalled in ATR for Cycle 154B, a new moisture filter was installed. Moisture data from Cycle 152B are Failed. All moisture data from the final three cycles (Cycles 154B, 155A, and 155B) are Qualified for use by the VHTR TDO Program.
The Graphite Technology Development Program will run a series of six experiments to quantify the effects of irradiation on nuclear-grade graphite. The third experiment, Advanced Graphite Creep 3 (AGC-3), began with Advanced Test Reactor (ATR) Cycle 152B on November 27, 2012, and ended with ATR Cycle 155B on April 23, 2014. This report documents qualification of AGC-3 experiment irradiation monitoring data for use by the Very High Temperature Reactor (VHTR) Technology Development Office (TDO) Program for research and development activities required to design and license the first VHTR nuclear plant. Qualified data meet the requirements for data collection and use as described in the experiment planning and quality assurance documents. Failed data do not meet the requirements. Trend data may not meet the requirements, but may still provide some useable information. The report documents qualification of AGC-3 experiment irradiation monitoring data following MCP-2691. This report also documents whether AGC-3 experiment irradiation monitoring data meet the requirements for data collection as specified in technical and functional requirements documents and quality assurance (QA) plans. Data handling is described showing how data are passed from the data collection experiment to the Nuclear Data Management and Analysis System (NDMAS) team. The data structure is described, including data batches, components, attributes, and response variables. The description of the approach to data qualification includes the steps taken to qualify the data and the specific tests used to verify that the data meet requirements. Finally, the current status of the data received by NDMAS from the AGC-3 experiment is presented with summarized information on test results and resolutions. This report addresses all of the irradiation monitoring data collected during the AGC-3 experiment.
Simple/Quick metered data processing method that can be used for Army Metered Data Management System (MDMS) and Logistics Innovation Agency data, but may also be useful for other large data sets. Intended for large data sets when analyst has little information about the buildings.
The data of a nuclear power plant (NPP) is stored in silos for the various plant tools. A previous study conducted by the LWRS program identified the different data sources that can benefit online monitoring in an NPP. These data have different structures and tools and are therefore used independently. Integrating data from various sources can result in multiple benefits to online monitoring by enhancing the ability to compare data, reduce statistical uncertainty due to a larger data set, increase data heterogeneity in space and time to reveal confounded information, develop intelligent methods, and share data. Data integration can result in direct cost saving by reducing the amount of effort needed to find needed information. It can also enable establishing a data warehouse, the backbone repository of data for an NPP, and is necessary to evolve NPPs to a holistic online monitoring approach and automate labor-intensive activities. Whether within an NPP or for a fleet of NPPs, deploying a data warehouse in a plant or data broker for a fleet of plants requires overcoming multiple challenges, including the lack of a standard form of storing and using data and the lack of integration methods to compare and map the different data sources. This effort demonstrates two methods and applications for data integration through an application integration pilot and a data fusion pilot. The application integration is targeted towards the software interfacing of data and data management. This approach is commonly used by data integrators. The data fusion pilot is targeted towards intelligently mapping or coupling data for optimized data use. The pilots were performed in collaboration with two NPPs.
Data compression methods for ITOS VHRR data were studied for a tape recorder record-and playback application. A playback period of 9 minutes was assumed with a nominal 18 minute record period for a 2-to-1 compression ratio. Both analog and digital methods were considered with the conclusion that digital methods should be used. Two system designs were prepared. One is a PCM system and the other is an entropy-coded predictive-quantization, sometimes called entropy-coded DPCM or just DPCM, system. Both systems use data management principles to transmit only the necessary data. Both systems use a medium capacity standard tape recorder from specifications provided by the technical officer. The 10 to the 9th power bit capacity of the recorder is the basic limitation on the compression ratio. Both systems achieve the minimum desired 2 to 1 compression ratio. A slower playback rate can be used with the DPCM system due to a higher compression factor for better link performance at a given CNR in terms of bandwidth utilization and error rate. The report is divided into two parts. The first part summarizes the theoretical conclusions of the second part and presents the system diagrams. The second part is a detailed analysis based upon an empirically derived random process model arrived at from specifications and measured data provided by the technical officer.
The Advanced Technology Laboratory (ATL) utilizes the Space Shuttle and the European Spacelab and is intended for advanced space technology. The evolution of the ATL concept is outlined along with its current status. Spacelab payload point design studies are reviewed as related to objective; physical layout; mission analysis; pointing accuracy; attitude management and fuel consumption; controls, displays, and data management; and thermal control. These studies aim at increasing the knowledge of the best techniques for integration and operation of payloads in the Spacelab and the Shuttle. ATL presents a dynamic new research potential, capable of accomodating 8 to 15 experiments per mission.
The processing of synthetic aperture radar (SAR) signals using the massively parallel processor (MPP) is discussed. The fast Fourier transform convolution procedures employed in the algorithms are described. The MPP architecture comprises an array unit (ARU) which processes arrays of data; an array control unit which controls the operation of the ARU and performs scalar arithmetic; a program and data management unit which controls the flow of data; and a unique staging memory (SM) which buffers and permutes data. The ARU contains a 128 by 128 array of bit-serial processing elements (PE). Two-by-four surarrays of PE's are packaged in a custom VLSI HCMOS chip. The staging memory is a large multidimensional-access memory which buffers and permutes data flowing with the system. Efficient SAR processing is achieved via ARU communication paths and SM data manipulation. Real time processing capability can be realized via a multiple ARU, multiple SM configuration.