AFS and DFS Growth at JPL
The goal for EIS File Service is to develop and deploy a service which encourages all JPL employees and contractors to use DFS to effiently share files, publish web pages, & access tools.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
The goal for EIS File Service is to develop and deploy a service which encourages all JPL employees and contractors to use DFS to effiently share files, publish web pages, & access tools.
NASA has recently deployed a new mid-range scheduling system for the antennas of the Deep Space Network (DSN), called Service Scheduling Software, or S(sup 3). This system was designed and deployed as a modern web application containing a central scheduling database integrated with a collaborative environment, exploiting the same technologies as social web applications but applied to a space operations context. This is highly relevant to the DSN domain since the network schedule of operations is developed in a peer-to-peer negotiation process among all users of the DSN. These users represent not only NASA's deep space missions, but also international partners and ground-based science and calibration users. The initial implementation of S(sup 3) is complete and the system has been operational since July 2011. This paper describes some key aspects of the S(sup 3) system and on the challenges of modeling complex scheduling requirements and the ongoing extension of S(sup 3) to encompass long-range planning, downtime analysis, and forecasting, as the next step in developing a single integrated DSN scheduling tool suite to cover all time ranges.
The high reliability required for Aeronautical components is a major reason for extensive Nondestructive Testing and Evaluation. Here at Langley Research Center (LaRC), there are highly trained and certified personal to conduct such testing to prevent hazards from occurring in the workplace and on the research projects for the National Aeronautics and Space Administration (NASA). The purpose of my studies was to develop a communication source to educate others of the services and equipment offered here. This was accomplished by creating documents that are accessible to all in the industry via the World Wide Web.
Effective midleware can improve the capability of business and science applications in several ways, e.g., by hiding platform heterogeneity or by providing standard shared services which reduce the complexity or increase the capability of every application. Recent successes in midleware, such as multi-tier client/server and web-based architectures, have fueled phenomenal growth in enterprise level applications, which provide better integration and more rapid adaptability of business in many fields.
Conditions data infrastructure for both ATLAS and CMS have to deal with the management of several Terabytes of data. Distributed computing access to this data requires particular care and attention to manage request-rates of up to several tens of kHz. Thanks to the large overlap in use cases and requirements, ATLAS and CMS have worked towards a common solution for conditions data management with the aim of using this design for data-taking in Run 3. In the meantime other experiments, including NA62, have expressed an interest in this cross- experiment initiative. For experiments with a smaller payload volume and complexity, there is particular interest in simplifying the payload storage. The conditions data management model is implemented in a small set of relational database tables. A prototype access toolkit consisting of an intermediate web server has been implemented, using standard technologies available in the Java community. Access is provided through a set of REST services for which the API has been described in a generic way using standard Open API specications, implemented in Swagger. Such a solution allows the automatic generation of client code and server stubs and further allows changes in the backend technology transparently. An important advantage of using a REST API for conditions access is the possibility of caching identical URLs, addressing one of the biggest challenges that large distributed computing solutions impose on conditions data access, avoiding direct DB access by means of standard web proxy solutions.
This white paper describes a series of tests that were performed to determine if a neural network could learn patterns from a service that maintains a cache of routing decisions for the discovery of version information for discrete web-at-large URLs. Labeled training data was derived from a log file that records the by-archive availability of Memento availability for a given URL. Training data sets were generated from this log file on a by-archive basis (thus making it a binary classification problem). Each training data set consisted of equal numbers of hit and miss URLs, selected at random. The URLs were converted to a normalized numeric representation where each integer in a URL training vector represents a character in that URL. The corresponding label indicates whether or not Mementos were available for that URL in the selected archive. The training data matrix and the label vector became input to a neural network. A number of neural network architectures and network hyperparameters were explored, however the log entries themselves were used as-is, without any feature engineering, beyond the aforementioned normalization.
Geographically distributed project teams need an Internet based collaborative work environment or "Intranet." The Virtual Research Center (VRC) is an experimental Intranet server that combines several services such as desktop conferencing, file archives, on-line publishing, and security. Using the World Wide Web (WWW) as a shared space paradigm, the Graphical User Interface (GUI) presents users with images of a lunar colony. Each project has a wing of the colony and each wing has a conference room, library, laboratory, and mail station. In FY95, the VRC development team proved the feasibility of this shared space concept by building a prototype using a Netscape commerce server and several public domain programs. Successful demonstrations of the prototype resulted in approval for a second phase. Phase 2, documented by this report, will produce a seamlessly integrated environment by introducing new technologies such as Java and Adobe Web Links to replace less efficient interface software.
Networks of video cameras, meteorological sensors, and ancillary electronic equipment are under development in collaboration among NASA Ames Research Center, the Federal Aviation Administration (FAA), and the National Oceanic Atmospheric Administration (NOAA). These networks are to be established at and near airports to provide real-time information on local weather conditions that affect aircraft approaches and landings. The prototype network is an airport-approach-zone camera system (AAZCS), which has been deployed at San Francisco International Airport (SFO) and San Carlos Airport (SQL). The AAZCS includes remotely controlled color video cameras located on top of SFO and SQL air-traffic control towers. The cameras are controlled by the NOAA Center Weather Service Unit located at the Oakland Air Route Traffic Control Center and are accessible via a secure Web site. The AAZCS cameras can be zoomed and can be panned and tilted to cover a field of view 220 wide. The NOAA observer can see the sky condition as it is changing, thereby making possible a real-time evaluation of the conditions along the approach zones of SFO and SQL. The next-generation network, denoted a remote tower sensor system (RTSS), will soon be deployed at the Half Moon Bay Airport and a version of it will eventually be deployed at Los Angeles International Airport. In addition to remote control of video cameras via secure Web links, the RTSS offers realtime weather observations, remote sensing, portability, and a capability for deployment at remote and uninhabited sites. The RTSS can be used at airports that lack control towers, as well as at major airport hubs, to provide synthetic augmentation of vision for both local and remote operations under what would otherwise be conditions of low or even zero visibility.
Power User Interface 5.0 (PUI) is a system of middleware, written for expert users in the Earth-science community, PUI enables expedited ordering of data granules on the basis of specific granule-identifying information that the users already know or can assemble. PUI also enables expert users to perform quick searches for orderablegranule information for use in preparing orders. PUI 5.0 is available in two versions (note: PUI 6.0 has command-line mode only): a Web-based application program and a UNIX command-line- mode client program. Both versions include modules that perform data-granule-ordering functions in conjunction with external systems. The Web-based version works with Earth Observing System Clearing House (ECHO) metadata catalog and order-entry services and with an open-source order-service broker server component, called the Mercury Shopping Cart, that is provided separately by Oak Ridge National Laboratory through the Department of Energy. The command-line version works with the ECHO metadata and order-entry process service. Both versions of PUI ultimately use ECHO to process an order to be sent to a data provider. Ordered data are provided through means outside the PUI software system.
The Astro2020 Decadal Survey recommended an investment in Time-Domain and 4 Multimessenger Astrophysics (TDAMM) as the top-priority sustaining activity in space for the 5 coming decade. One aspect of NASA’s response to this recommendation is a pilot project, 6 the Astrophysics Cross-Observatory Science Support (ACROSS) initiative, designed to provide 7 support to both missions and observers as they pursue TDAMM science. Here, we present our 8 observations of needs in the community and initial plans for ACROSS activities, including services 9 to facilitate and improve cross-mission follow-up planning and execution; a multimessenger web portal with links to existing mission resources, community tools, and information targeted for 11 TDAMM general observers; development of “Smart target-of-opportunity submission page” proof-12 of-concepts; and ongoing development of a potential TDAMM general observing competitive 13 grant solicitation. As the ACROSS pilot phase begins, we invite discussion of our plans with both 14 missions and observers to better understand their needs and concerns.
The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models feature diverse scientific workflows as simple as linear pipelines or as complex as a directed acyclic graphs, which must be executed and supported over wide-area networks with massively distributed resources. Application users oftentimes need to manually configure their computing tasks over networks in an ad hoc manner, hence significantly limiting the productivity of scientists and constraining the utilization of resources. The success of these large-scale distributed applications requires a highly adaptive and massively scalable workflow platform that provides automated and optimized computing and networking services. This project is to design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a web-based user interface specially tailored for a target application, a set of user libraries, and several easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in heterogeneous high-performance network environments. SWAMP will enable the automation and management of the entire process of scientific workflows with the convenience of a few mouse clicks while hiding the implementation and technical details from end users. Particularly, we will consider two types of applications with distinct performance requirements: data-centric and service-centric applications. For data-centric applications, the main workflow task involves large-volume data generation, catalog, storage, and movement typically from supercomputers or experimental facilities to a team of geographically distributed users; while for service-centric applications, the main focus of workflow is on data archiving, preprocessing, filtering, synthesis, visualization, and other application-specific analysis. We will conduct a comprehensive comparison of existing workflow systems and choose the best suited one with open-source code, a flexible system structure, and a large user base as the starting point for our development. Based on the chosen system, we will develop and integrate new components including a black box design of computing modules, performance monitoring and prediction, and workflow optimization and reconfiguration, which are missing from existing workflow systems. A modular design for separating specification, execution, and monitoring aspects will be adopted to establish a common generic infrastructure suited for a wide spectrum of science applications. We will further design and develop efficient workflow mapping and scheduling algorithms to optimize the workflow performance in terms of minimum end-to-end delay, maximum frame rate, and highest reliability. We will develop and demonstrate the SWAMP system in a local environment, the grid network, and the 100Gpbs Advanced Network Initiative (ANI) testbed. The demonstration will target scientific applications in climate modeling and high energy physics and the functions to be demonstrated include workflow deployment, execution, steering, and reconfiguration. Throughout the project period, we will work closely with the science communities in the fields of climate modeling and high energy physics including Spallation Neutron Source (SNS) and Large Hadron Collider (LHC) projects to mature the system for production use.
The ability to use data stored in the current NASA Earth Observing System (EOS) archives for studying regional or global phenomena is highly dependent on having a detailed understanding of the data's internal structure and physical implementation. Gaining this understanding and applying it to data reduction is a time-consuming task that must be undertaken before the core investigation can begin. This is an especially difficult challenge when science objectives require users to deal with large multi-sensor data sets that are usually of different formats, structures, and resolutions. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has taken a major step towards meeting this challenge by developing an infrastructure with a Web interface that allows users to perform interactive analysis online without downloading any data, the GES-DISC Interactive Online Visualization and Analysis Infrastructure or "Giovanni." Giovanni provides interactive, online, analysis tools for data users to facilitate their research. There have been several instances of this interface created to serve TRMM users, Aerosol scientists, Ocean Color and Agriculture applications users. The first generation of these tools support gridded data only. The user selects geophysical parameters, area of interest, time period; and the system generates an output on screen in a matter of seconds. The currently available output options are: Area plot averaged or accumulated over any available data period for any rectangular area; Time plot time series averaged over any rectangular area; Hovmoller plots image view of any longitude-time and latitude-time cross sections; ASCII output for all plot types; Image animation for area plot. Another analysis suite deals with parameter intercomparison: scatter plots, temporal correlation maps, GIs-compatible outputs, etc. This allow user to focus on data content (i.e. science parameters) and eliminate the need for expensive learning, development and processing tasks that are redundantly incurred by an archive's user community. The current implementation utilizes the GrADS-DODS Server (GDS), and provides subsetting and analysis services across the Internet for any GrADS-readable dataset. The subsetting capability allows users to retrieve a specified temporal and/or spatial subdomain from a large dataset, eliminating the need to download everything simply to access a small relevant portion of a dataset. The analysis capability allows users to retrieve the results of an operation applied to one or more datasets on the server. We use this approach to read pre-processed binary files and/or to read and extract the needed parts directly from HDF or HDF-EOS files. These subsets then serve as inputs into GrADS analysis scripts. It can be used in a wide variety of Earth science applications: climate and weather events study and monitoring; modeling. It can be easily configured for new applications.
Many early web-based digital libraries (DLs) had implicit assumptions reflected in their architecture that the unit of focus in the DL (frequently "reports" or "e-prints") would only be manifested in a single, or at most a few, common file formats such as PDF or PostScript. DLs have now matured to the point where their contents are commonly no longer simple files. Complex objects in DLs have emerged from in response to various requirements, including: simple aggregation of formats and supporting files, bundling additional information to aid digital preservation, creating opaque digital objects for e-commerce applications, and the incorporation of dynamic services with the traditional data files. We examine a representative (but not necessarily exhaustive) number of current and recent historical web-based complex object technologies and projects that are applicable to DLs: Aurora, Buckets, ComMentor, Cryptolopes, Digibox, Document Management Alliance, FEDORA, Kahn-Wilensky Framework Digital Objects, Metadata Encoding & Transmission Standard, Multivalent Documents, Open eBooks, VERS Encapsulated Objects, and the Warwick Framework.
The ability to use data stored in the current Earth Observing System (EOS) archives for studying regional or global phenomena is highly dependent on having a detailed understanding of the data's internal structure and physical implementation. Gaining this understanding and applying it to data reduction is a time- consuming task that must be undertaken before the core investigation can begin. This is an especially difficult challenge when science objectives require users to deal with large multi-sensor data sets that are usually of different formats, structures, and resolutions, for example, when preparing data for input into modeling systems. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has taken a major step towards meeting this challenge by developing an infrastructure with a Web interface that allows users to perform interactive analysis online without downloading any data, the GES-DISC Interactive Online Visualization and Analysis Infrastructure or "Giovanni." Giovanni provides interactive, online, analysis tools for data users to facilitate their research. There have been several instances of this interface created to serve TRMM users, Aerosol scientists, Ocean Color and Agriculture applications users. The first generation of these tools support gridded data only. The user selects geophysical parameters, area of interest, time period; and the system generates an output on screen in a matter of seconds. The currently available output options are: Area plot averaged or accumulated over any available data period for any rectangular area; Time plot time series averaged over any rectangular area; Time plots image view of any longitude-time and latitude-time cross sections; ASCII output for all plot types; Image animation for area plot. In the future, we will add correlation plots, GIS-compatible outputs, etc. This allow user to focus on data content (i.e. science parameters) and eliminate the need for expensive learning, development and processing tasks that are redundantly incurred by an archive's user community. The current implementation utilizes the GrADS-DODS Server (GDS), a stable, secure data server that provides subsetting and analysis services across the Internet for any GrADS-readable dataset. The subsetting capability allows users to retrieve a specified temporal and/or spatial subdomain from a large dataset, eliminating the need to download everything simply to access a small relevant portion of a dataset. The analysis capability allows users to retrieve the results of an operation applied to one or more datasets on the server. In our case, we use this approach to read pre-processed binary files and/or to read and extract the needed parts from HDF or HDF-EOS files. These subsets then serve as inputs into GrADS processing and analysis scripts. It can be used in a wide variety of Earth science applications: climate and weather events study and monitoring; modeling. It can be easily configured for new applications.
The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), in collaboration with NASA Sounder Team at the Jet Propulsion Laboratory (JPL), provides processing, archiving, and distribution services for remotely-sensed data acquired by satellite sounders. Supported data sets begin chronologically with the legacy TIROS Operational Vertical Sounder (TOVS) Pathfinder, continue to the Atmospheric Infrared Sounder (AIRS), a modern hyperspectral sounder onboard the Aqua satellite, and are followed by data from the subsequent Suomi-National Polar-orbiting Partnership Cross-track Infrared Sounder (CrIS) mission and the Joint Polar Satellite System (JPSS) series CrIS missions. These satellite sounders provide long-term global observations of the atmospheric state, including temperature and humidity profiles, outgoing longwave radiation, cloud properties, and trace gases. Applications of sounder data products cover a broad range of fields, including meteorology climatology, hydrology, and air quality. The GES DISC has developed many services to assist users, including simplified and efficient methods for searching, accessing, downloading, and analytically exploring these satellite sounder data products. We have also developed the Giovanni system, a broadly used Web-based application, which provides a simple and intuitive way to visualize, analyze, and access Earth science remote sensing data. In this presentation, we will introduce the standard and near-real time sounder data products, and demonstrate our services through some use cases. Highlights of our service capabilities include data subset, vertical profile plot, inter-comparison, multi-year monthly/seasonal mean, interannual monthly/seasonal time series, and anomaly analysis.
Space weather models are essential to our ability to understand and predict space weather events. For over 20 years, the Community Coordinated Modeling Center (CCMC, https://ccmc.gsfc.nasa.gov) has been providing transformative tools and platforms for hosting space weather models and associated services, free and open to anyone interested in studying space weather. Runs-on-Request system (ROR) is one of the popular services at CCMC that permits researchers and other end-users to exercise cutting-edge hosted heliophysics and space weather models using a simple web interface, as well as collaborate on an extensive and continuously growing archive of over 28,000 model run results. Similar to other projects at CCMC, ROR has grown as a community project that strives to be open and transparent to its users. In this poster, we discuss some of our recent efforts to further expose ROR data, metadata, and services to the end users through both custom and community-developed access protocols. We also discuss how in-house science support provided by the CCMC team plays a paramount role in making ROR data and services truly accessible by the community.
The alignment and calibration workflows at the Compact Muon Solenoid (CMS) experiment are fundamental to provide a high quality physics data and to maintain the design performance of the experiment. To facilitate the operational efforts required by the experiment, the alignment and calibration team has developed and deployed a set of web-based applications to search, navigate and prepare a consistent set of calibrations to be consumed in reconstruction of data for physics, accessible through the Condition DB Browser. The Condition DB Browser hosts also various data management tools, including a visualization tool that allows to easily inspect alignment an calibration contents, an user-defined notification agent for delivering updates on modification to the database, a logging service for the user and the automatic online-to-offline condition uploads. In this paper we report on the operational experience of this web application from 2017 data taking, with focus on new features and tools incorporated during this period.
The Urban Environment Initiative (UEI), has been established as part of a Cooperative Agreement with the National Aeronautics and Space Administration (NASA). The UEI is part of NASA's overall High Performance Computing and Communications (HPCC) and the Information Infrastructure Technology Applications (IITA) programs. The goal of the UEI is to provide public access to Earth Science information and promote its use with a focus on the environment of urban areas. This goal will be accomplished through collaborative efforts of the UEI team with both community-based and local/regional governmental organizations. The UEI team is comprised of four organizations representing private industry, NASA, and universities: Prime Technologies Service Corporation, NASA's Minority University Space Interdisciplinary Network (MU-SPIN) California State University, at Los Angeles, and Central State University (Wilberforce, OH). "Urban Environment" refers to the web of environmental, economic, and social factors that combine to create the urban world in which we live. Examples of these factors are population distribution, neighborhood demographic profiles, economic resources, business activities, location and concentration of environmental hazards and various pollutants, proximity and level of urban services, which form the basis of the urban environment and ultimately affect our lives and experiences. The use of Geographic Information Systems (GIS) and remote sensing allows data to be visualized in the forms of maps and spatial images. The use of these tools allow analysis of information about urban environments. Also included are descriptions of the four query types which will assist in understanding the maps.