Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “csv”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32

Automated Metadata Extraction: Challenges and Opportunities

Proper application of the FAIR data principles is what separates a vibrant data ecosystem, in which research data are frequently shared and reused, from a lifeless data graveyard. Automated metadata extraction systems have been proposed as a means of bolstering the findability, interoperability, and reusabil- ity of data repositories with little or no human intervention. These extraction systems mine metadata by crawling a repository and applying lightweight extractors that, for various types of file (e.g., image, CSV file), extract or synthesize relevant attributes. In practice, however, the automated creation of generally useful metadata is fraught with challenges. Data consumers may have different perspectives as to what metadata representations are useful, the standards for recording metadata tend to change over time, and the software model for processing updates can introduce unnecessary human and computational effort. Thus, generalizing extraction for a broad audience of data consumers is a difficult and relatively unsolved problem.In this work, we explore these challenges faced by extraction systems in the context of constructing our own extraction system for science data. We first define the metadata extraction problem and provide context to the issues faced in generalizing metadata. Additionally, we identify potential research directions to help alleviate many of these challenges for all automated extraction systems. Ultimately, this work represents a first step in designing ubiquitous metadata extraction systems that can maximize the value of research data while minimizing the human efforts required in doing so.

Skluzacek, Tyler↗

A Scientific Data Provenance Harvester for Distributed Applications

Data provenance provides a way for scientists to observe how experimental data originates, conveys process history, and explains influential factors such as experimental rationale and associated environmental factors from system metrics measured at runtime. The US Department of Energy Office of Science Integrated end-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows (IPPD) project has developed a provenance harvester that is capable of collecting observations from file based evidence typically produced by distributed applications. To achieve this, file based evidence is extracted and transformed into an intermediate data format inspired in part by W3C CSV on the Web recommendations, called the Harvester Provenance Application Interface (HAPI) syntax. This syntax provides a general means to pre-stage provenance into messages that are both human readable and capable of being written to a provenance store, Provenance Environment (ProvEn). HAPI is being applied to harvest provenance from climate ensemble runs for Accelerated Climate Modeling for Energy (ACME) project funded under the U.S. Department of Energy’s Office of Biological and Environmental Research (BER) Earth System Modeling (ESM) program. ACME informally provides provenance in a native form through configuration files, directory structures, and log files that contain success/failure indicators, code traces, and performance measurements. Because of its generic format, HAPI is also being applied to harvest tabular job management provenance from Belle II DIRAC scheduler relational database tables as well as other scientific applications that log provenance related information.

Stephan, Eric G.↗

Tundra Trait Team: A database of plant traits spanning the tundra biome

Motivation: The Tundra Trait Team (TTT) database includes field-based measurements of key traits related to plant form and function at multiple sites across the tundra biome. This dataset can be used to address theoretical questions about plant strategy and trade-offs, trait-environment relationships and environmental filtering, and trait variation across spatial scales, to validate satellite data, and to inform earth system model parameters. Main types of variable contained: The database contains 81,537 measurements of 18 plant traits. The most frequently measured traits (>1,000 observations each) include plant height, leaf area, specific leaf area, leaf fresh and dry mass, leaf dry matter content, leaf nitrogen content, leaf carbon content, leaf phosphorus content, seed mass, and stem specific density. Spatial location and grain: Measurements were collected in tundra habitats in both the northern and southern hemispheres, including Arctic sites in Alaska, Canada, Greenland, Fennoscandia, and Siberia and alpine sites in the European Alps, Colorado Rockies, Caucasus, Ural Mountains, Pyrenees, Australian Alps, and Central Otago Mountains (New Zealand). More than 99% of observations are georeferenced. Time period and grain: All data were collected between 1964 and 2016. A small number of sites have repeated trait measurements at two or more time periods. Major taxa and level of measurement: Trait measurements were made on 858 terrestrial vascular plant species growing in tundra habitats. Most observations are on individuals (87%), while a small number represent plot or site means or maximums per species. Software format: csv file and GitHub repository with data cleaning scripts in R; contribution to TRY plant trait database (www.try-db.org) to be included in the next version release.

59 BASIC BIOLOGICAL SCIENCES↗

STANDARDS-BASED TECHNOLOGIES FOR EXCHANGING FATIGUE TEST DATA

Fatigue testing campaigns are a common feature in the design and operation of advanced engineering systems in the aerospace and power generation sectors. The resulting data are typically of a high inherent technical and financial value. Presently, these data are typically transferred between departments and companies by way of ad-hoc solutions reliant on obsolete or proprietary technologies, including CSV files, MS Excel® files, and PDFs. In these circumstances there is significant potential for data loss, inconsistency, and error. To address these shortcomings, there is a need for a systematic means of transferring data between different digital systems. With this in mind, a series of CEN Workshops on engineering materials data have taken place with a view to developing technologies for representing and exchanging engineering materials data. Most recently, a CEN Workshop on the topic of fatigue test data has delivered data formats derived from the ISO 12106 standard for axial strain-controlled fatigue testing. This paper describes the methodology for developing the data formats and demonstrates their use in the scope of the INCEFA PLUS project on increasing safety in nuclear power plants by covering gaps in environmental fatigue assessment.

Austin, Timothy↗

SU-E-T-641: Development and Verification of Automatic Reading Dose of Interest From Eclipse's DVH

Purpose: According to clinical and research requirement, we develop a function of automatic reading dose of interest from dose volume histogram(DVH), to replace the traditional method with a mouse one by one point, and it's also verified. Methods: The DVH automatic reading function will be developed in an in-house developed radiotherapy information management system(RTIMS), which is based on Apache+PHP+MySQL. A DVH ASCII file is exported from Varian Eclipse V8.6, which includes the following contents: 1. basic information of patient; 2. dose information of plan; 3. dose information of structures, including basic information and dose volume data of target volume and organ at risk. And the default exported dose volume data also includes relative doses by 1% step and corresponding absolute doses and cumulative relative volumes, and the volumes are 4 decimal fraction. Clinically, we often need read the doses of some integer percent volumes, such as D50 and D30. So it couldn't be directly obtained from the above data, but we can use linear interpolation bye the near volumes and doses: Dx=D2−(V2−Vx)*(D2−D1)/(V2−V1), and program a function to search, read and calculate the corresponding data. And the doses of all preseted volume of interest of all structures can be automatically read one by one patient, and saved as a CSV file. To verify it, we select 24 IMRT plans for prostate cancer, and doses of interest are PTV D98/D95/D5/D2, bladder D30/D50, and rectum D25/D50. Two groups of data, using the automatic reading method(ARM) and pointed dose method(PDM), are analyzed with SPSS 16. The absolute difference=D-ARM-D-PDM, relative difference=absolute difference*100%/prescription dose(7600cGy). Results: The differences are as following: PTV D98/D95/D5/D2: −0.04%/− 0.04%/0.13%/0.19%, bladder D30/D50: −0.02%/0.01%, and rectum D25/D50: 0.03%/0.01%. Conclusion: Using this function, the error is very small, and can be neglected. It could greatly improve the efficiency of clinical work. Project supported by the National Natural Science Foundation of China (Grant No.81101694)

60 APPLIED LIFE SCIENCES↗

SU-E-T-241: Design and Implement of An Information Management System for Radiation Workers in a Hospital

Purpose: To meet the special needs of Medical Administration Division, an information management system for radiation workers in a hospital(IMSRWH) has been developed. Methods: There are about 200 radiation workers in 20 departments, such as department of radiology, department of radiation oncology, department of nuclear medicine, and so on. An IMSRWH server was used to run a database and web service of Apache+PHP+MySQL. The IMSRWH web service could be accessed with Internet Explorer(IE) to input, search, count, and print the radiation workers’ data. Every worker’s data information includes the following tables: registration, personal dose monitoring records, personnel certificate, training certificate, personnel license, medical examination, etc, and each table includes more optional detail information. Results: In the past year, the IMSRWH has been successfully developed and implemented in the hospital. According to the needs, the user rights are divided into: the system administrator, office manager, department manager and radiation worker. The different users have the different interfaces, and the appropriate authority permissions mean corresponding operations. The administrator and office manager could manage all personnel information, department manager could manage the staff information of his department, and the workers could view or modify their own information. Because information updates, involving hundreds of each entry and modification are complicated, each table information of all workers can be batch import from or export to a Microsoft Excel CSV format file, to update all together. However, it has only Chinese version now. Conclusion: It was demonstrated to be user-friendly and was proven to significantly improve the efficiency of the hospital management. Since it is an in-house developed system, more functions can be added or modified to further enhance its potentials in research and clinical practice.

60 APPLIED LIFE SCIENCES↗

SU-F-J-100: Standardized Biodistribution Template for Nuclear Medicine Dosimetry Collection and Reporting

Purpose: As the field of Nuclear Medicine moves forward with efforts to integrate radiation dosimetry into clinical practice we can identify the challenge posed by the lack of standardized dose calculation methods and protocols. All personalized internal dosimetry is derived by projecting biodistribution measurements into dosimetry calculations. In an effort to standardize organization of data and its reporting, we have developed, as a sequel to the EANM recommendation of “Good Dosimetry Reporting”, a freely available biodistribution template, which can be used to create a common point of reference for dosimetry data. It can be disseminated, interpreted, and used for method development widely across the field. Methods: A generalized biodistribution template was built in a comma delineated format (.csv) to be completed by users performing biodistribution measurements. The template is available for free download. The download site includes instructions and other usage details on the template. Results: This is a new resource developed for the community. It is our hope that users will consider integrating it into their dosimetry operations. Having biodistribution data available and easily accessible for all patients processed is a strategy for organizing large amounts of information. It may enable users to create their own databases that can be analyzed for multiple aspects of dosimetry operations. Furthermore, it enables population data to easily be reprocessed using different dosimetry methodologies. With respect to dosimetry-related research and publications, the biodistribution template can be included as supplementary material, and will allow others in the community to better compare calculations and results achieved. Conclusion: As dosimetry in nuclear medicine become more routinely applied in clinical applications, we, as a field, need to develop the infrastructure for handling large amounts of data. Our organ level biodistribution template can be used as a standard format for data collection, organization, as well as for dosimetry research and software development.

60 APPLIED LIFE SCIENCES↗

SU-G-TeP4-08: Automating the Verification of Patient Treatment Parameters

Purpose: To automate the daily verification of each patient’s treatment by utilizing the trajectory log files (TLs) written by the Varian TrueBeam linear accelerator while reducing the number of false positives including jaw and gantry positioning errors, that are displayed in the Treatment History tab of Varian’s Chart QA module. Methods: Small deviations in treatment parameters are difficult to detect in weekly chart checks, but may be significant in reducing delivery errors, and would be critical if detected daily. Software was developed in house to read TLs. Multiple functions were implemented within the software that allow it to operate via a GUI to analyze TLs, or as a script to run on a regular basis. In order to determine tolerance levels for the scripted analysis, 15,241 TLs from seven TrueBeams were analyzed. The maximum error of each axis for each TL was written to a CSV file and statistically analyzed to determine the tolerance for each axis accessible in the TLs to flag for manual review. The software/scripts developed were tested by varying the tolerance values to ensure veracity. After tolerances were determined, multiple weeks of manual chart checks were performed simultaneously with the automated analysis to ensure validity. Results: The tolerance values for the major axis were determined to be, 0.025 degrees for the collimator, 1.0 degree for the gantry, 0.002cm for the y-jaws, 0.01cm for the x-jaws, and 0.5MU for the MU. The automated verification of treatment parameters has been in clinical use for 4 months. During that time, no errors in machine delivery of the patient treatments were found. Conclusion: The process detailed here is a viable and effective alternative to manually checking treatment parameters during weekly chart checks.

60 APPLIED LIFE SCIENCES↗

Observation of anomalous thermal Hall effect in a Kagome superconductor

Broken time-reversal symmetry (TRS) in superconductors can induce not only spontaneous magnetization by the finite angular momentum of Cooper pairs but also the anomalous thermal Hall effects (ATHEs), whose detection has been extremely challenging. Here, we report the successful observation of an ATHE developing below the superconducting transition temperature at zero magnetic field in the kagome-lattice superconductor CsV 3 Sb 5 . This finding is verified by the absence of a signal in a conventional type-II superconductor using the same setup and by ruling out the trapped-vortex effects through micro-Hall array measurements. Both the temperature dependence and the magnitude of the observed anomalous thermal Hall conductivity are quite different from those expected for the quantized thermal edge current of an intrinsic ATHE but consistent with extrinsic impurity-induced ATHEs in chiral superconductivity. Our study of ATHE offers an alternative approach to probe TRS breaking in the superconducting states.

Yoshida, Hiroki [Univ. of Tokyo, Chiba (Japan)] (O↗

2019 Annual Technology Baseline ATB Cost and Performance Data for Electricity Generation Technologies

Starting in 2015 NREL has presented the Annual Technology Baseline ATB in an Excel workbook that contains detailed cost and performance data both current and projected for renewable and conventional technologies. The workbook includes a spreadsheet for each technology. This updated version of the workbook provides data for the 2019 ATB. This year NREL is also providing a packaged Tableau workbook and structured summary csv files. A website documents this data at https://atb.nlr.gov .

13 HYDRO ENERGY↗

LCLS RF Station Anomaly Candidates, SLAC, Nov 2020 to Dec 2020, Version 1

The data consists of radio-frequency (RF) station process variables (amplitude and amplitude-mean-out-of-tolerance bit) and beam position monitor (BPM) values for about a month of Linac Coherent Light Source (LCLS) time. The data is all from normal conducting hard X-ray operation. The dataset contains roughly 5000 anomaly “candidates.” For each candidate, we include 3.5 minutes of data from the 82 RF stations and ~20 seconds of data from 4 BPMs. We also include several CSV files with additional information and labels for each candidate.

43 PARTICLE ACCELERATORS↗

Dataset for the paper titled "Investigating the relationship between bolide entry angle and apparent direction of infrasound signal arrivals"

This dataset includes outputs generated for the journal publication titled: "Investigating the relationship between bolide entry angle and apparent direction of infrasound signal arrivals". The outputs include .csv files with model-generated synthetic trajectories of asteroids entering Earth at a variety of impact and approach (azimuthal) angles. All outputs are based on hypothetical but realistic scenarios.

Herrera, Natalie [Sandia National Laboratories (SN↗

PylotDB - A Database Management, Graphing, and Analysis Tool Written in Python

PylotDB, written completely in Python, provides a user interface (UI) with which to interact with, analyze, graph data from, and manage open source databases such as MySQL. The UI mitigates the user having to know in-depth knowledge of the database application programming interface (API). PylotDB allows the user to generate various kinds of plots from user-selected data; generate statistical information on text as well as numerical fields; backup and restore databases; compare database tables across different databases as well as across different servers; extract information from any field to create new fields; generate, edit, and delete databases, tables, and fields; generate or read into a table CSV data; and similar operations. Since much of the database information is brought under control of the Python computer language, PylotDB is not intended for huge databases for which MySQL and Oracle, for example, are better suited. PylotDB is better suited for smaller databases that might be typically needed in a small research group situation. PylotDB can also be used as a learning tool for database applications in general.

Barnette, DanielW.↗

NGDS Data Archiver

This is a Node.js command line utility for scraping XML metadata from CSW and WFS, downloading linkage data from CSW and WFS, pinging hosts and returning status codes, pinging data linkages and returning status codes, writing ping status to CSV files, and uploading data to Amazon S3.

Source record↗

GDX-pandas [SWR 15-21]

This is a Python package to translate between gdx (GAMS data) and pandas. There are two main ways to use gdxpds. The first use case is the one that was initially supported: direct conversion between GDX files on disk and pandas DataFrames or a csv version thereof. Starting with the Version 1.0.0 rewrite, there is now a second style of use which involves interfacing with GDX files and symbols via the `gdxpds.gdx.GdxFile` and `gdxpds.gdx.GdxSymbol` classes. Please visit https://nrel.github.io/gdx-pandas for the latest documentation.

Hale, Elaine↗

PLEXOS Input Data Generator

The PLEXOS Input Data Generator (PIDG) is a tool that enables PLEXOS users to better version their data, automate data processing, collaborate in developing inputs, and transfer data between different production cost modeling and other power systems analysis software. PIDG can process data that is in a generalized format from multiple input sources, including CSV files, PostgreSQL databases, and PSS/E .raw files and write it to an Excel file that can be imported into PLEXOS with only limited manual intervention.

Source record↗

ThermalTracker

ThermalTracker tracks birds and bats in thermal video. It will process video in any standard format that was recorded with a thermal video camera. The output is images of the flight tracks of birds and bats in PNG format, and a comma-separated value (CSV) file listing the details of the extracted flight tracks.

Matzner, Shari↗

Crosslinking IMS

Given a list of proteins, finds all cross-linked peptides in the given Ion Mobility MS dataset. This application attempts to find all cross-linked peptides in the given Ion Mobility MS dataset. First, the FASTA file is read in and possible peptides are generated by digesting the protein(s) and assuming 1 max missed cleavage. The possible peptides are searched for in the given features by matching up the monoisotopic mass of the feature to the mass of the peptides. The mass ppm tolerance is an input parameter of the application. All possible cross-links of the unmodified peptides are then calculated and searched for in the Isotopic Peak data. A detailed csv results file is created at the end.

Monroe, Matthew↗