Final Report - "GSIMF: A Grid Software Installation Management Framework for HEP Data Analysis"
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Since the beginning of the National Spherical Torus Experiment's (NSTX) research activities in 1999, open-source and collaborative software have been used extensively for experimental operations. The software included Experimental Physics and Industrial Control System (EPICS) and the Model Data System plus (MDSplus). This paper will begin with a retrospective of ten years of integrated computing, control, and data analysis on NSTX. A variety of practical issues will be reviewed such as: system reliability, availability, and maintainability; using open source software with commercialoff- the-shelf (COTS) hardware; obsolescence and remaining current. Recent improvements, current status, and emerging issues will be presented. Numerous proposals for NSTX upgrades have been made that will extend the performance and operating lifetime of NSTX. In light of this, a collective strategy for improving NSTX's collaborative software model for a new decade of operations will be examined.
The IRIS-HEP software institute, as a contributor to the broader HEP Python ecosystem, is developing scalable analysis infrastructure and software tools to address the upcoming HL-LHC computing challenges with new approaches and paradigms, driven by our vision of what HL-LHC analysis will require. The institute uses a "Grand Challenge" format, constructing a series of increasingly large, complex, and realistic exercises to show the vision of HL-LHC analysis. Recently, the focus has been demonstrating the IRIS-HEP analysis infrastructure at scale and evaluating technology readiness for production. As a part of the Analysis Grand Challenge activities, the institute executed a "200 Gbps Challenge", aiming to show sustained data rates into the event processing of multiple analysis pipelines. The challenge integrated teams internal and external to the institute, including operations and facilities, analysis software tools, innovative data delivery and management services, and scalable analysis infrastructure. The challenge showcases the prototypes - including software, services, and facilities - built to process around 200 TB of data in both the CMS NanoAOD and ATLAS PHYSLITE data formats with test pipelines. The teams were able to sustain the 200 Gbps target across multiple pipelines. The pipelines focusing on event rate were able to process at over 30 MHz. These target rates are demanding; the activity revealed considerations for future testing at this scale and changes necessary for physicists to work at this scale in the future. The 200 Gbps Challenge has established a baseline on today's facilities, setting the stage for the next exercise at twice the scale.
The paper presents a case study of the interannual variability of vertical Ekman transport (upwelling) observed in the North Atlantic during the CZCS data collection period in order to illustrate SEAPAK's ancillary data handling functions. The study begins at the entire basin-scale for the North Atlantic and proceeds to a regional scale where much of the observed upwelling variability occurs. The analysis utilizes a variety of output types, including images, horizontal contour plots, time series, cross sections, and vertical profiles, produced on the VAX directly or by commercially available PC software packages. The SEAPAK software can be used just as effectively for global or basin-scale studies as for regional studies.
This thesis consists of two parts that have contributed to a new meson spectroscopy program, MesonEx, which is currently taking place in Hall B of the upgraded Thomas Jefferson National Accelerator Facility (JLab) in the USA. The first, hardware part, presents design, testing, construction and calibration of a fast-timing scintillation Hodoscope for a new Forward Tagger detector, that has been installed inside the upgraded CEBAF Large Acceptance Spectrometer (CLAS12) which is situated in the Hall B of JLab. The Forward Tagger is a key apparatus for measurement of quasi-real photoproduction of mesons in the MesonEx, and a necessary device for other new hadron spectroscopy programs. The second, software and analysis part, presents contributions to the general HAdron SPEctroscopy CenTre (HASPECT) analysis framework, that has been developed in preparation for the MesonEx data analysis. The software contribution consists of finalizing HASPECT simulation and analysis chains in a model-independent way, and developing a mass-independent partial wave analysis procedure. This procedure has been tested via analysis of the γ p → pK + K - channel from the g 11 a CLAS data, in the photon energy range E γ = 3.0-3.8 GeV and momentum transfer squared range -t = 0.6 - 0.7 (GeV/c 2 ) 2 . The first result of this analysis is the differential cross section for the ϕ(1020) resonance photoproduction. This result has been compared with a previous analysis result, and a good agreement has validated the developed analysis procedure. Furthermore, S , P and D partial waves have been extracted from the data set, using the same procedure, and ambiguous solutions for these partial waves have been calculated, using the method of Barrelet zeros which is for the first time applied to photoproduction of the K + and K - mesons on the proton. Distributions of the calculated solutions have been compared with the fit results. It is found that the physical solution contains contributions of the ϕ(1020) in the P -wave and the a 2(1320) in the D -wave.
NASA Langley Research Center's (LaRC) High Intensity Radiated Fields Laboratory (HIRF Lab) recently conducted a series of electromagnetic radiated emissions tests under a cooperative agreement with Delta Airlines and an interagency agreement with the FAA. The frequency spectrum environment at a commercial airport was measured on location. The environment survey provides a comprehensive picture of the complex nature of the electromagnetic environment present in those areas outside the aircraft. In addition, radiated emissions tests were conducted on portable electronic devices (PEDs) that may be brought onboard aircraft. These tests were performed in both semi-anechoic and reverberation chambers located in the HIRF Lab. The PEDs included cell phones, laptop computers, electronic toys, and family radio systems. The data generated during the tests are intended to support the research on the effect of radiated emissions from wireless devices on aircraft systems. Both tests systems relied on customized control and data reduction software to provide test and instrument control, data acquisition, a user interface, real time data reduction, and data analysis. The software executed on PC's running MS Windows 98 and 2000, and used Agilent Pro Visual Engineering Environment (VEE) development software, Common Object Model (COM) technology, and MS Excel.
A software method has been developed that is applicable for analyzing cylindrical and partially cylindrical objects inspected using computed tomography (CT). This method involves unwrapping and re-slicing data so that the CT data from the cylindrical object can be viewed as a series of 2D sheets (or flattened onion skins ) in addition to a series of top view slices and 3D volume rendering. The advantages of viewing the data in this fashion are as follows: (1) the use of standard and specialized image processing and analysis methods is facilitated having 2D array data versus a volume rendering; (2) accurate lateral dimensional analysis of flaws is possible in the unwrapped sheets versus volume rendering; (3) flaws in the part jump out at the inspector with the proper contrast expansion settings in the unwrapped sheets; and (4) it is much easier for the inspector to locate flaws in the unwrapped sheets versus top view slices for very thin cylinders. The method is fully automated and requires no input from the user except proper voxel dimension from the CT experiment and wall thickness of the part. The software is available in 32-bit and 64-bit versions, and can be used with binary data (8- and 16-bit) and BMP type CT image sets. The software has memory (RAM) and hard-drive based modes. The advantage of the (64-bit) RAM-based mode is speed (and is very practical for users of 64-bit Windows operating systems and computers having 16 GB or more RAM). The advantage of the hard-drive based analysis is one can work with essentially unlimited-sized data sets. Separate windows are spawned for the unwrapped/re-sliced data view and any image processing interactive capability. Individual unwrapped images and un -wrapped image series can be saved in common image formats. More information is available at http://www.grc.nasa.gov/WWW/OptInstr/ NDE_CT_CylinderUnwrapper.html.
Context. Ground-based γ-ray astronomy is still a rather young field of research, with strong historical connections to particle physics. This is why most observations are conducted by experiments with proprietary data and analysis software, as is usual in the particle physics field. However, in recent years, this paradigm has been slowly shifting toward the development and use of open-source data formats and tools, driven by upcoming observatories such as the Cherenkov Telescope Array (CTA). In this context, a community-driven, shared data format (the gamma-astro-data-format, or GADF) and analysis tools such as Gammapy and ctools have been developed. So far, these efforts have been led by the Imaging Atmospheric Cherenkov Telescope community, leaving out other types of ground-based γ-ray instruments. Aims. We aim to show that the data from ground particle arrays, such as the High-Altitude Water Cherenkov (HAWC) observatory, are also compatible with the GADF and can thus be fully analyzed using the related tools, in this case, Gammapy. Methods. We reproduced several published HAWC results using Gammapy and data products compliant with GADF standard. We also illustrate the capabilities of the shared format and tools by producing a joint fit of the Crab spectrum including data from six different γ-ray experiments. Results. We find excellent agreement with the reference results, a powerful confirmation of both the published results and the tools involved. Conclusions. The data from particle detector arrays such as the HAWC observatory can be adapted to the GADF and thus analyzed with Gammapy. A common data format and shared analysis tools allow multi-instrument joint analysis and effective data sharing. To emphasize this, a sample of Crab nebula event lists is made public with this paper. Because of the complementary nature of pointing and wide-field instruments, this synergy will be distinctly beneficial for the joint scientific exploitation of future observatories such as the Southern Wide-field Gamma-ray Observatory and CTA.
The WETO Stack software is a collection of data, data analysis scripts, and website content produced through the Holistic Modeling Project’s Portfolio Coordination task. The purpose of the software is to characterize the collection of wind energy software projects supported by the U.S. Department of Energy’s Wind Energy Technologies Office. The included data contains characteristics of the actively funded software projects, and the data analysis scripts provide insights into the breadth, depth, and maturity of the collection of software projects. The website content presents the results of the analyses as well as reports from workshops and other events. It can be viewed at https://nrel.github.io/WETOStack/index.html.
Digital security camera technologies proliferate the security marketplace with manufacturers extoling the functions, features, specifications and virtues of their cameras. A process to comparatively analyze security camera technologies using international camera analysis standards and methods provides a defensible means to assess camera image quality on a technically comparable basis. The procedures in this document use ISO standard camera test charts and camera image analysis software to perform test chart image analysis for comparative camera assessment. Step-by-step procedures for acquiring test chart images in the laboratory and producing analysis data using analysis software GUI controls and interfaces for comparative analysis of security camera image quality are detailed in this document. The procedures have been validated through the test and analysis of fixed box and 180 degree field of view multi-imager cameras as well as, fixed and varifocal lenses and analog-to-digital video encoder image quality performance attributes.
RT-Display is a MATLAB-based data acquisition environment designed to use a variety of commercial off-the-shelf (COTS) hardware to digitize analog signals to a standard data format usable by other post-acquisition data analysis tools. This software presents the acquired data in real time using a variety of signal-processing algorithms. The acquired data is stored in a standard Operator Interactive Signal Processing Software (OISPS) data-formatted file. RT-Display is primarily configured to use the Agilent VXI (or equivalent) data acquisition boards used in such systems as MIDDAS (Multi-channel Integrated Dynamic Data Acquisition System). The software is generalized and deployable in almost any testing environment, without limitations or proprietary configuration for a specific test program or project. With the Agilent hardware configured and in place, users can start the program and, in one step, immediately begin digitizing multiple channels of data. Once the acquisition is completed, data is converted into a common binary format that also can be translated to specific formats used by external analysis software, such as OISPS and PC-Signal (product of AI Signal Research Inc.). RT-Display at the time of this reporting was certified on Agilent hardware capable of acquisition up to 196,608 samples per second. Data signals are presented to the user on-screen simultaneously for 16 channels. Each channel can be viewed individually, with a maximum capability of 160 signal channels (depending on hardware configuration). Current signal presentations include: time data, fast Fourier transforms (FFT), and power spectral density plots (PSD). Additional processing algorithms can be easily incorporated into this environment.
The modern aberration-corrected scanning electron microscopes have successfully achieved direct visualization of atomic columns with sub-angstrom resolution. With this significant progress, advanced image quantification and analysis is still at its early stages. In this work, we present the complete pathway for the metrology of atomic resolution STEM images. This includes: 1) tips for acquiring high-quality STEM images; 2) denoising and drift-correction for enhancing measurement accuracy; 3) obtaining initial atom positions; 4) indexing the atoms based on unit cell vectors; 4) quantifying the atom column positions with either 2D-gaussian single peak fitting or 5) multi-peak fitting routines for slightly overlapping atomic columns; 6) quantification of lattice distortion/strain within the crystal structures or at the defects/interfaces where the lattice periodicity is disrupted, and 7) some common methods to visualize and present the analysis. Furthermore, a simple self-developed free MATLAB app (EASY-STEM) with a graphical user interface (GUI) will be introduced that can help with the analysis of STEM images without the need for writing dedicated analysis code or software. The advanced data analysis methods presented here can be applied for the local quantification of defect relaxations, local structural distortions, local phase transformations, and non-centrosymmetry in a wide range of materials.
The high-luminosity program has seen numerous extrapolations of its needed computing resources that each indicate the need for substantial changes if the desired HL-LHC physics program is to be supported within the current level of computing resource budgets. Drivers include large increases in event complexity (leading to increased processing time and analysis data size) and trigger rates needed (5-10 fold increases) for the HL-LHC program. The CMS experiment has recently undertaken an effort to merge the ideas behind short-term and long-term resource models in order to make easier and more reliable extrapolations to future needs. Near term computing resource estimation requirements depend on numerous parameters: LHC uptime and beam intensities; detector and online trigger performance; software performance; analysis data requirements; data access, management, and retention policies; site characteristics; and network performance. Longer term modeling is affected by the same characteristics, but with much larger uncertainties that must be considered to understand the most interesting handles for increasing the "physics per computing dollar" of the HL-LHC. In this presentation, we discuss the current status of long term modeling of the CMS computing resource needs for HL-LHC with emphasis on techniques for extrapolations, uncertainty quantification, and model results. We illustrate potential ways that high-luminosity CMS could accomplish its desired physics program within today's computing budgets.
This software package runs the atmospheric model MM5 in data assimilation cycling mode to produce an optimized weather analysis, including the ability to insert or adjust a hurricane vortex. The program runs MM5 through a cycle of short forecasts every three hours where the vortex is adjusted to match the observed hurricane location and storm intensity. This technique adjusts the surrounding environment so that the proper steering current and environmental shear are achieved. MM5cycle uses a Cressman analysis to blend observation into model fields to get a more accurate weather analysis. Quality control of observations is also done in every cycle to remove bad data that may contaminate the analysis. This technique can assimilate and propagate data in time from intermittent and infrequent observations while maintaining the atmospheric field in a dynamically balanced state. The software consists of a C-shell script (MM5cycle.driver) and three FORTRAN programs (splitMM5files.F, comRegrid.F, and insert_vortex.F), and are contained in the pre-processor component of MM5 called "Regridder." The model is first initialized with data from a global model such as the Global Forecast System (GFS), which also provides lateral boundary conditions. These data are separated into single-time files using splitMM5.F. The hurricane vortex is then bogussed in the correct location and with the correct wind field using insert_vortex.F. The modified initial and boundary conditions are then recombined into the model fields using comRegrid.F. The model then makes a three-hour forecast. The three-hour forecast data from MM5 now become the analysis for the next short forecast run, where the vortex will again be adjusted. The process repeats itself until the desired time of analysis is achieved. This code can also assimilate observations if desired.
This SBIR effort was focused on developing a production ready system to address the integration and interoperability challenges with analysis and visualization in fusion simulations. Our overarching technical objective was to minimize the code development simulation scientists incur when coupling their simulation codes with different analysis frameworks. To this end, we developed an open-source software library to make data exchange between application easier and a Web application to manage and display analysis extracts from simulations. We have also augmented existing libraries funded by DOE such as ADIOS and VTK-m. When used together, these make it significantly easier to integrate simulation and analysis capability. We demonstrated the flexibility of our approach using two common simulation codes in the fusion community, XGC1 and GTC.
These software release notes for QuickView Software will be used as an appendix in a user manual (which will also be STIPped when completed).
Ambulatory physiological monitoring has been used to study human health and performance in space and in a variety of Earth-based environments (e.g., military aircraft, armored vehicles, small groups in isolation, and patients). Large, multi-channel data files are typically recorded in these environments, and these files often require the removal of contaminated data prior to processing and analyses. Physiological data processing can now be performed with user-friendly, interactive software developed by the Ames Psychophysiology Research Laboratory. This software, which runs on a Windows platform, contains various signal-processing routines for both time- and frequency- domain data analyses (e.g., peak detection, differentiation and integration, digital filtering, adaptive thresholds, Fast Fourier Transform power spectrum, auto-correlation, etc.). Data acquired with any ambulatory monitoring system that provides text or binary file format are easily imported to the processing software. The application provides a graphical user interface where one can manually select and correct data artifacts utilizing linear and zero interpolation and adding trigger points for missed peaks. Block and moving average routines are also provided for data reduction. Processed data in numeric and graphic format can be exported to Excel. This software, PostProc (for post-processing) requires the Dadisp engineering spreadsheet (DSP Development Corp), or equivalent, for implementation. Specific processing routines were written for electrocardiography, electroencephalography, electromyography, blood pressure, skin conductance level, impedance cardiography (cardiac output, stroke volume, thoracic fluid volume), temperature, and respiration
Ion mobility (IM) is often combined with LC-MS experiments to provide an additional dimension of separation for complex sample analysis. While highly complex samples are better characterized by the full dimensionality of LC-IM-MS experiments to uncover new information, downstream data analysis workflows are often not equipped to properly mine the additional IM dimension. For many samples the data acquisition benefits of including IM separations are all that is necessary to uncover sample information and the full dimensionality of the data is not required for data analysis. Post-acquisition reduction and adaptation of the dimensions of LC-IM-MS and IM-MS experiments into an LC-MS format opens the possibility to use a plethora of existing software tools. In this work, we developed data file conversion tools to reduce the complexity of IM data analysis. Three data file transformations are introduced in the PNNL PreProcessor software: 1) mapping the IM axis to the LC axis for IM-MS data, 2) converting the drift time vs. m/z space to CCS/z vs m/z space, and 3) transforming All Ions IM/MS mobility aligned fragmentation data to a standard LC-MS DDA data file format. Finally, these new data file conversions are demonstrated with corresponding lipidomics and proteomics workflows that leverage existing LC-MS data analysis software to highlight the benefits of the data transformations.