Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Automatic Data Processing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52
All sourcesNASA NTRSDOE OSTI
NASA NTRS1990

Numerical derivative techniques for trajectory optimization

The adoption of robust numerical optimization techniques in trajectory simulation programs has resulted in powerful design and analysis tools. These trajectory simulation/optimization programs are widely used, and a representative list includes the GTS system, the POST program, and newer collocation methods such as OTIS and FONPAC. All of these programs rely on optimization algorithms which require objective function and constraint gradient data during the iteration process. However, most trajectory optimization problems lack simple analytical expressions for these derivatives. In the general case a function evaluation involves integrating aerodynamic, propulsive, and gravity forces over multiple trajectory phases with complex control models. With the newer collocation methods, the integration is replaced by defect constraints and cubic approximations for the state. While analytic gradient expressions can sometimes be derived for trajectory optimization problems, the derivation is cumbersome, time consuming, and prone to mistakes. Fortunately, an alternate method exists for the gradient evaluation, namely finite difference approximations. In this paper some finite difference gradient techniques developed for use with the GTS system are presented. These techniques include methods for computing first and second partial derivatives of single and multiple sets of functions. A key feature of these methods is an error control mechanism which automatically adjusts the perturbation size to obtain accurate derivative values.

Hallman, Wayne P.↗
NASA NTRS2004

Knowledge Driven Image Mining with Mixture Density Mercer Kernels

This paper presents a new methodology for automatic knowledge driven image mining based on the theory of Mercer Kernels; which are highly nonlinear symmetric positive definite mappings from the original image space to a very high, possibly infinite dimensional feature space. In that high dimensional feature space, linear clustering, prediction, and classification algorithms can be applied and the results can be mapped back down to the original image space. Thus, highly nonlinear structure in the image can be recovered through the use of well-known linear mathematics in the feature space. This process has a number of advantages over traditional methods in that it allows for nonlinear interactions to be modelled with only a marginal increase in computational costs. In this paper, we present the theory of Mercer Kernels, describe its use in image mining, discuss a new method to generate Mercer Kernels directly from data, and compare the results with existing algorithms on data from the MODIS (Moderate Resolution Spectral Radiometer) instrument taken over the Arctic region. We also discuss the potential application of these methods on the Intelligent Archive, a NASA initiative for developing a tagged image data warehouse for the Earth Sciences.

Srivastava, Ashok N.↗
NASA NTRS2004

Knowledge Driven Image Mining with Mixture Density Mercer Kernals

This paper presents a new methodology for automatic knowledge driven image mining based on the theory of Mercer Kernels, which are highly nonlinear symmetric positive definite mappings from the original image space to a very high, possibly infinite dimensional feature space. In that high dimensional feature space, linear clustering, prediction, and classification algorithms can be applied and the results can be mapped back down to the original image space. Thus, highly nonlinear structure in the image can be recovered through the use of well-known linear mathematics in the feature space. This process has a number of advantages over traditional methods in that it allows for nonlinear interactions to be modelled with only a marginal increase in computational costs. In this paper we present the theory of Mercer Kernels; describe its use in image mining, discuss a new method to generate Mercer Kernels directly from data, and compare the results with existing algorithms on data from the MODIS (Moderate Resolution Spectral Radiometer) instrument taken over the Arctic region. We also discuss the potential application of these methods on the Intelligent Archive, a NASA initiative for developing a tagged image data warehouse for the Earth Sciences.

Srivastava, Ashok N.↗
NASA NTRS1992

The evolution of the FIGARO data reduction system

The Figaro data reduction system originated at Caltech around 1983. It was based on concepts being developed in the U.K. by the Starlink organization, particularly the use of hierarchical self-defining data structures and the abstraction of most user-interaction into a set of 'parameter system' routines. Since 1984 it has continued to be developed at AAO, in collaboration with Starlink and Caltech. It was adopted as Starlink's main spectroscopic data reduction package, although it is by no means limited to spectra; it has operations for images and data cubes and even a few (very specialized) for four-dimensional data hypercubes. It continued to be used at Caltech and will be used at the Keck. It is also in use at a variety of other organizations around the world. Figaro was originally a system for VMS Vaxes. Recently it was ported (at Caltech) to run on SUN's, and work is underway at the University of New South Wales on a DecStation version. It is hoped to coordinate all this work into a unified release, but coordination of the development of a system by organizations covering three continents poses a number of interesting administrative problems. The hierarchical data structures used by Figaro allow it to handle a variety of types of data, and to add new items to data structures. Error and data quality information was added to the basic file format used, error information being particularly useful for infrared data. Cooperating sets of programs can add specific sub-structures to data files to carry information that they understand (polarimetry data containing multiple data arrays, for example), without this affecting the way other programs handle the files. Complex instrument-specific ancillary information can be added to data files written at a telescope and can be used by programs that understand the instrumental details in order to produce properly calibrated data files. Once this preliminary data processing was done the resulting files contain 'ordinary' spectra or images that can be processed by programs that are not instrument-specific. The structures holding the instrumental information can then be discarded from the files. Much effort has gone into trying to make it easy to write Figaro programs; data access subroutines are now available to handle access to all the conventional items found in Figaro files (main data arrays, error information, quality information etc), and programs that only need to access such items can be very simple indeed. A large number of Figaro users do indeed write their own Figaro applications using these routines. The fact that Figaro programs are written as callable subroutines getting information from the user through a small set of parameter routines means that they can be invoked in numerous ways; they are normally linked and run as individual programs (called by a small main routine that is generated automatically), but are also available linked to run under the ADAM data acquisition system and there is an interface that lets them be called as part of a user-written Fortran program. The long-term future of Figaro probably depends to a large extent on how successfully it manages the transition from being a VMS-only system to being a multi-platform system.

Shortridge, K.↗
NASA NTRS1998

Resiliency of the Multiscale Retinex Image Enhancement Algorithm

The multiscale retinex with color restoration (MSRCR) continues to prove itself in extensive testing to be very versatile automatic image enhancement algorithm that simultaneously provides dynamic range compression, color constancy, and color rendition, However, issues remain with regard to the resiliency of the MSRCR to different image sources and arbitrary image manipulations which may have been applied prior to retinex processing. In this paper we define these areas of concern, provide experimental results, and, examine the effects of commonly occurring image manipulation on retinex performance. In virtually all cases the MSRCR is highly resilient to the effects of both the image source variations and commonly encountered prior image-processing. Significant artifacts are primarily observed for the case of selective color channel clipping in large dark zones in a image. These issues are of concerning the processing of digital image archives and other applications where there is neither control over the image acquisition process, nor knowledge about any processing done on th data beforehand.

Rahman, Zia-Ur↗
NASA NTRSDate not supplied

Development of the GeoNEX Level 2G Products: Exploiting the Diurnal Variability of TOA Reflectance in Atmospheric Correction

This study develops a new atmospheric correction algorithm to generate the Level 2G products, in particular the gap-filled Surface Reflectance at 10-minute time steps, for the Geostationary-NASA Earth Exchange (GeoNEX) project. The algorithm is based on the MODIS MAIAC (Multi-Angle Implementation of Atmospheric Correction) framework but with significant modifications to exploit angular/temporal information from the diurnal variability of the GeoNEX L1G TOA (Top-of-Atmosphere) reflectance. The algorithm starts by evaluating the roughness/smoothness of the diurnal time series of the TOA reflectance. Because rapid changes in TOA reflectance are generally caused by passing clouds or shadows, rough segments of the time series are automatically filtered out while the smooth segments are further tested for brightness and temperature to identify clear-sky and snow-free observations. Next the algorithm runs the MAIAC RTM (Radiative Transfer Model) to retrieve the Ross-Thick-Li-Sparse (RTLS) BRDF model parameters and the daily-mean atmospheric optical depth (AOD) that allow the RTM to optimally simulate the observed diurnal variability of clear-sky TOA reflectance. Once the initial RTLS parameters are retrieved after the algorithm’s burn-in period, they are used as the prior information to predict the AOD level for the next days, while the subsequent clear-sky observations are used to make necessary adjustments to the RTLS parameters in an continuous fashion. This “prediction-analysis” cycle is then iterated to process the full time series of the L1G data, skipping only total-cloudy days or when surface snow is detected. We tested the algorithm over a list of selected AERONET sites. The retrieved results (the daily mean AOD and the RTLS parameters) reasonably agree with the ground-based measurements. Importantly, the results indicate that the diurnal cycles of surface reflectance are continuous functions of the illumination-view geometry. Thus we can use the retrieved RTLS model to accurately fill in data gaps on partial cloudy days. Also, our algorithm is totally independent from the traditional approaches based on the use of spectral band ratios between the shortwave infrared (e.g., 2.2µm) and the visible (e.g., 0.47µm and 0.64µm) bands. Our results thus demonstrate that the high-frequent diurnal geostationary observations contain unique information that helps us improve atmospheric correction of remote sensing data.

GeoNEX↗
NASA NTRS2008

Displaying Composite and Archived Soundings in the Advanced Weather Interactive Processing System

In a previous task, the Applied Meteorology Unit (AMU) developed spatial and temporal climatologies of lightning occurrence based on eight atmospheric flow regimes. The AMU created climatological, or composite, soundings of wind speed and direction, temperature, and dew point temperature at four rawinsonde observation stations at Jacksonville, Tampa, Miami, and Cape Canaveral Air Force Station, for each of the eight flow regimes. The composite soundings were delivered to the National Weather Service (NWS) Melbourne (MLB) office for display using the National version of the Skew-T Hodograph analysis and Research Program (NSHARP) software program. The NWS MLB requested the AMU make the composite soundings available for display in the Advanced Weather Interactive Processing System (AWIPS), so they could be overlaid on current observed soundings. This will allow the forecasters to compare the current state of the atmosphere with climatology. This presentation describes how the AMU converted the composite soundings from NSHARP Archive format to Network Common Data Form (NetCDF) format, so that the soundings could be displayed in AWl PS. The NetCDF is a set of data formats, programming interfaces, and software libraries used to read and write scientific data files. In AWIPS, each meteorological data type, such as soundings or surface observations, has a unique NetCDF format. Each format is described by a NetCDF template file. Although NetCDF files are in binary format, they can be converted to a text format called network Common data form Description Language (CDL). A software utility called ncgen is used to create a NetCDF file from a CDL file, while the ncdump utility is used to create a CDL file from a NetCDF file. An AWIPS receives soundings in Binary Universal Form for the Representation of Meteorological data (BUFR) format (http://dss.ucar.edu/docs/formats/bufr/), and then decodes them into NetCDF format. Only two sounding files are generated in AWIPS per day. One file contains all of the soundings received worldwide between 0000 UTC and 1200 UTC, and the other includes all soundings between 1200 UTC and 0000 UTC. In order to add the composite soundings into AWIPS, a procedure was created to configure, or localize, AWIPS. This involved modifying and creating several configuration text files. A unique fourcharacter site identifier was created for each of the 32 soundings so each could be viewed separately. The first three characters were based on the site identifier of the observed sounding, while the last character was based on the flow regime. While researching the localization process for soundings, the AMU discovered a method of archiving soundings so old soundings would not get purged automatically by AWl PS. This method could provide an alternative way of localizing AWl PS for composite soundings. In addition, this would allow forecasters to use archived soundings in AWIPS for case studies. A test sounding file in NetCDF format was written in order to verify the correct format for soundings in AWIPS. After the file was viewed successfully in AWIPS, the AMU wrote a software program in the Tool Command Language/Tool Kit (Tcl/Tk) language to convert the 32 composite soundings from NSHARP Archive to CDL format. The ncgen utility was then used to convert the CDL file to a NetCDF file. The NetCDF file could then be read and displayed in AWIPS.

Barrett, Joe H., III↗
NASA NTRS2001

Left ventricular endocardial surface detection based on real-time 3D echocardiographic data

OBJECTIVE: A new computerized semi-automatic method for left ventricular (LV) chamber segmentation is presented. METHODS: The LV is imaged by real-time three-dimensional echocardiography (RT3DE). The surface detection model, based on level set techniques, is applied to RT3DE data for image analysis. The modified level set partial differential equation we use is solved by applying numerical methods for conservation laws. The initial conditions are manually established on some slices of the entire volume. The solution obtained for each slice is a contour line corresponding with the boundary between LV cavity and LV endocardium. RESULTS: The mathematical model has been applied to sequences of frames of human hearts (volume range: 34-109 ml) imaged by 2D and reconstructed off-line and RT3DE data. Volume estimation obtained by this new semi-automatic method shows an excellent correlation with those obtained by manual tracing (r = 0.992). Dynamic change of LV volume during the cardiac cycle is also obtained. CONCLUSION: The volume estimation method is accurate; edge based segmentation, image completion and volume reconstruction can be accomplished. The visualization technique also allows to navigate into the reconstructed volume and to display any section of the volume.

Non-NASA Center↗
NASA NTRS1976

Chemiluminescence Study on Thermal Degradation of Aircraft Tire Elastomers

Since the autoxidative process accounts in part for the degradation of rubber, including aircraft tires, it was felt that a study of the chemiluminescence from unsaturated elastomers could contribute significantly to an understanding of the degradation mechanism. The study revealed similarities in chemiluminescence behavior between four elastomers which were investigated, and it shows that similar oxidation mechanisms occur. Oxidative chemiluminescence was observed from purified samples of cis-1,4-polybutadiene, cis-1,4-polyisoprene, trans-polypentenamer, and 1,2-polybutadiene in an oxygen atmosphere at 25-150 C. The elastomer samples were placed in a 600 watt oven which is equipped with gas inlets for introducing any desired atmosphere. Chemiluminescence emission from the samples was focused with a two inch quartz lens onto the detector of a 12" photomultiplier which is connected to a photon counter. A strip-chart recorder, connected to the counter, permitted automatic data collection. Diagrams of the apparatus are included. The chemical reactions which occurred from the thermal decomposition of the polymer samples are described, and results (and tabulated data) are discussed.

Mendenhall, G. D.↗
NASA NTRS2018

Changes in Exercise Data Management

The suite of exercise hardware aboard the International Space Station (ISS) generates an immense amount of data. The data collected from the treadmill, cycle ergometer, and resistance strength training hardware are basic exercise parameters (time, heart rate, speed, load, etc.). The raw data are post processed in the laboratory and more detailed parameters are calculated from each exercise data file. Updates have recently been made to how this valuable data are stored, adding an additional level of data security, increasing data accessibility, and resulting in overall increased efficiency of medical report delivery. Questions regarding exercise performance or how exercise may influence other variables of crew health frequently arise within the crew health care community. Inquiries over the health of the exercise hardware often need quick analysis and response to ensure the exercise system is operable on a continuous basis. Consolidating all of the exercise system data in a single repository enables a quick response to both the medical and engineering communities. A SQL server database is currently in use, and provides a secure location for all of the exercise data starting at ISS Expedition 1 - current day. The database has been structured to update derived metrics automatically, making analysis and reporting available within minutes of dropping the inflight data it into the database. Commercial tools were evaluated to help aggregate and visualize data from the SQL database. The Tableau software provides manageable interface, which has improved the laboratory's output time of crew reports by 67%. Expansion of the SQL database to be inclusive of additional medical requirement metrics, addition of 'app-like' tools for mobile visualization, and collaborative use (e.g. operational support teams, research groups, and International Partners) of the data system is currently being explored.

Buxton, R. E.↗
NASA NTRS2012

Improving the Automatic Inversion of Digital Alouette/ISIS Ionogram Reflection Traces into Topside Electron Density Profiles

The topside sounders of the International Satellites for Ionospheric Studies (ISIS) program were designed as analog systems. The resulting ionograms were displayed on 35 mm film for analysis by visual inspection. Each of these satellites, launched between 1962 and 1971, produced data for 10 to 20 years. A number of the original telemetry tapes from this large data set have been converted directly into digital records. Software, known as the Topside Ionogram Scalar With True-Height (TOPIST) algorithm, has been produced and used for the automatic inversion of the ionogram reflection traces on more than 100,000 ISIS-2 digital topside ionograms into topside vertical electron density profiles Ne(h). Here we present some topside ionospheric solar cycle variations deduced from the TOPIST database to illustrate the scientific benefit of improving and expanding the topside ionospheric Ne(h) database. The profile improvements will be based on improvements in the TOPIST software motivated by direct comparisons between TOPIST profiles and profiles produced by manual scaling in the early days of the ISIS program. The database expansion will be based on new software designed to overcome limitations in the original digital topside ionogram database caused by difficulties encountered during the analog-to-digital conversion process in the detection of the ionogram frame sync pulse and/or the frequency markers. This improved and expanded TOPIST topside Ne(h) database will greatly enhance investigations into both short- and long-term ionospheric changes, e.g., the observed topside ionospheric responses to magnetic storms, induced by interplanetary magnetic clouds, and solar cycle variations, respectively.

Benson, Robert F.↗
NASA NTRS2018

Changes in Exercise Data Management

The suite of exercise hardware aboard the International Space Station (ISS) generates an immense amount of data. The data collected, treadmill, cycle ergometer, and resistance strength training hardware, are basic exercise parameters (time, heart rate, speed, load, etc.). The raw data are processed in the laboratory and more detailed parameters are calculated from each exercise data file. Updates recently have been made to how these valuable data are stored, adding an additional level of security, increasing accessibility, and resulting in overall increased efficiency of medical report delivery. Questions regarding exercise performance or how exercise may influence other variables of crew health frequently arise within the crew health care community. Inquiries regarding the health of the exercise hardware often need quick analysis and response to ensure the exercise system is operable on a continuous basis. Consolidating all of the exercise system data in a single repository enables a quick response to both the medical and engineering communities. A SQL server database is currently in use, and provides a secure location for all of the exercise data starting at ISS Expedition 1 to current date. The database has been structured to update derived metrics automatically, making analysis and reporting available within minutes of dropping the in-flight data into the database. Commercial tools were evaluated to help aggregate and visualize data from the SQL database. The Tableau software provides manageable interface, which has improved the laboratory’s output time of crew reports by 67%. Expansion of the SQL database, to be inclusive of additional medical requirement metrics, addition of ‘app-like’ tools for mobile visualization, and collaborative use (e.g., operational support teams, research groups, and International Partners) of the data system, is currently being explored.

Buxton, R. E.↗
NASA NTRS2020

Leveraging Google Earth Engine User Interface for Semiautomated Wetland Classification in the Great Lakes Basin at 10 m With Optical and Radar Geospatial Datasets

As one of the world’s largest freshwater ecosystems,the Great Lakes Basin houses hundreds of thousands of acres of wetlands that support a variety of crucial ecological and environmental functions at the local, regional, and global levels.Monitoring these wetlands is critical to conservation and restoration efforts, however current methods that rely on field monitoring are labor-intensive, costly, and often outdated. In this study, we present a graphical user interface constructed in Google Earth Engine called the Wetland Extent Tool (WET),which allows semi-automatic wetland classification according to a user-input area of interest and date range. WET composites datasets and conducts multi source, moderate resolution processing utilizing Landsat 8 OLI, Sentinel-2 MSI, Sentinel-1 C-SAR, and Shuttle Radar Topography Mission (SRTM) datasets to classify wetlands in the entire Great Lakes Basin. We evaluated classification results of wetlands, uplands, and open water from May-September 2019, and tested whether SRTM elevation, slope,or the Dynamic Surface Water Extent produced the most accurate results in each Great Lake Basin in conjunction with optical indices and radar composites. We found that elevation produced the most accurate classification in Lake Erie, Michigan,and Ontario, while slope performed best in Lake Huron and Superior. Lake Erie, Michigan, Ontario, and Huron achieved high overall accuracy and identification of wetlands. WET leverages cloud-computing for multi source processing of moderate resolution remote sensing data, and employs a user interface in Google Earth Engine that wetland managers and conservationists can use to monitor wetland extent in the Great Lakes Basin in near real-time.

Vanessa L Valenti↗
NASA NTRS2005

An XML Representation for Crew Procedures

NASA ensures safe operation of complex systems through the use of formally-documented procedures, which encode the operational knowledge of the system as derived from system experts. Crew members use procedure documentation on the ground for training purposes and on-board space shuttle and space station to guide their activities. Investigators at JSC are developing a new representation for procedures that is content-based (as opposed to display-based). Instead of specifying how a procedure should look on the printed page, the content-based representation will identify the components of a procedure and (more importantly) how the components are related (e.g., how the activities within a procedure are sequenced; what resources need to be available for each activity). This approach will allow different sets of rules to be created for displaying procedures on a computer screen, on a hand-held personal digital assistant (PDA), verbally, or on a printed page, and will also allow intelligent reasoning processes to automatically interpret and use procedure definitions. During his NASA fellowship, Dr. Simpson examined how various industries represent procedures (also called business processes or workflows), in areas such as manufacturing, accounting, shipping, or customer service. A useful method for designing and evaluating workflow representation languages is by determining their ability to encode various workflow patterns, which depict abstract relationships between the components of a procedure removed from the context of a specific procedure or industry. Investigators have used this type of analysis to evaluate how well-suited existing workflow representation languages are for various industries based on the workflow patterns that commonly arise across industry-specific procedures. Based on this type of analysis, it is already clear that existing workflow representations capture discrete flow of control (i.e., when one activity should start and stop based on when other activities start and stop), but do not capture the flow of data, materials, resources or priorities. Existing workflow representation languages are also limited to representing sequences of discrete activities, and cannot encode procedures involving continuous flow of information or materials between activities.

Simpson, Richard C.↗
NASA NTRS1994

Directional emittance surface measurement system and process

Apparatus and process for measuring the variation of directional emittance of surfaces at various temperatures using a radiometric infrared imaging system. A surface test sample is coated onto a copper target plate provided with selective heating within the desired incremental temperature range to be tested and positioned onto a precision rotator to present selected inclination angles of the sample relative to the fixed positioned and optically aligned infrared imager. A thermal insulator holder maintains the target plate on the precision rotator. A screen display of the temperature obtained by the infrared imager, and inclination readings are provided with computer calculations of directional emittance being performed automatically according to equations provided to convert selected incremental target temperatures and inclination angles to relative target directional emittance values. The directional emittance of flat black lacquer and an epoxy resin measurements obtained are in agreement with the predictions of the electromagnetic theory and with directional emittance data inferred from directional reflectance measurements made on a spectrophotometer.

Puram, Chith K.↗
NASA NTRS2007

Potential Space Applications for Body-Centric Wireless and E-Textile Antennas

Space environment benefits of body-centric wireless communications are numerous, particularly in the context of long duration Lunar and Martian outposts that are in planning stages at several space agencies around the world. Since crew time for such missions is a scarce commodity, seamless integration of body-centric wireless from various sources is paramount. Sources include traditional data, such as audio, video, tracking, and biotelemetry. Newer data sources include positioning, orientation, and status of handheld tools and devices, as well as management and status of on-body inventories. In addition to offering lighter weight and flexibility, performance benefits of e-textile antennas are anticipated due to advantageous use of the body s surface area. In creating e-textile antennas and RF devices, researchers are faced with the challenge of transferring conventional and novel designs to textiles. Lack of impedance control, limited conductivity, and the inability to automatically create intricate designs are examples of limitations frequently attributed to e-textiles. Reliable interfaces between e-textiles and conventional hardware also represent significant challenges. Addressing these limitations is critical to the continued development and acceptance of fabric-based circuits for body-centric wireless applications. Here we present several examples of e-textile antennas and RF devices, created using a NASA-developed process, that overcome several of these limitations. The design and performance of an equiangular spiral, miniaturized spiral-loaded slot antenna, and a hybrid coupler are considered, with the e-textile devices showing comparable performance to like designs using conventional materials.

Kennedy, Timothy F.↗
NASA NTRS1987

Proof that timing requirements of the FDDI token ring protocol are satisfied

The fiber distributed data interface (FDDI) is an ANSI draft proposed standard for a 100 Mbit/s fiber-optic token ring. The FDDI timed token access protocol provides dynamic adjustment of the load offered to the ring, with the goal of maintaining a specified token rotation time and of providing a guaranteed upper bound on time between successive arrivals of the token at a station. FDDI also provides automatic recovery when errors occur. The bound on time between successive token arrivals is guaranteed only if the token rotates quickly enough to satisfy timer requirements in each station when all ring resources are functioning properly. Otherwise, recovery would be initiated unnecessarily. The purpose of this paper is to prove that FDDI timing requirements are satisfied, i.e., the token rotates quickly enough to prevent initiation of recovery unless there is failure of a physical resource or unless the network management entity within a station initiates the recovery process.

Johnson, Marjory J.↗
NASA NTRS1997

Cooperative Data Sharing: Simple Support for Clusters of SMP Nodes

Libraries like PVM and MPI send typed messages to allow for heterogeneous cluster computing. Lower-level libraries, such as GAM, provide more efficient access to communication by removing the need to copy messages between the interface and user space in some cases. still lower-level interfaces, such as UNET, get right down to the hardware level to provide maximum performance. However, these are all still interfaces for passing messages from one process to another, and have limited utility in a shared-memory environment, due primarily to the fact that message passing is just another term for copying. This drawback is made more pertinent by today's hybrid architectures (e.g. clusters of SMPs), where it is difficult to know beforehand whether two communicating processes will share memory. As a result, even portable language tools (like HPF compilers) must either map all interprocess communication, into message passing with the accompanying performance degradation in shared memory environments, or they must check each communication at run-time and implement the shared-memory case separately for efficiency. Cooperative Data Sharing (CDS) is a single user-level API which abstracts all communication between processes into the sharing and access coordination of memory regions, in a model which might be described as "distributed shared messages" or "large-grain distributed shared memory". As a result, the user programs to a simple latency-tolerant abstract communication specification which can be mapped efficiently to either a shared-memory or message-passing based run-time system, depending upon the available architecture. Unlike some distributed shared memory interfaces, the user still has complete control over the assignment of data to processors, the forwarding of data to its next likely destination, and the queuing of data until it is needed, so even the relatively high latency present in clusters can be accomodated. CDS does not require special use of an MMU, which can add overhead to some DSM systems, and does not require an SPMD programming model. unlike some message-passing interfaces, CDS allows the user to implement efficient demand-driven applications where processes must "fight" over data, and does not perform copying if processes share memory and do not attempt concurrent writes. CDS also supports heterogeneous computing, dynamic process creation, handlers, and a very simple thread-arbitration mechanism. Additional support for array subsections is currently being considered. The CDS1 API, which forms the kernel of CDS, is built primarily upon only 2 communication primitives, one process initiation primitive, and some data translation (and marshalling) routines, memory allocation routines, and priority control routines. The entire current collection of 28 routines provides enough functionality to implement most (or all) of MPI 1 and 2, which has a much larger interface consisting of hundreds of routines. still, the API is small enough to consider integrating into standard os interfaces for handling inter-process communication in a network-independent way. This approach would also help to solve many of the problems plaguing other higher-level standards such as MPI and PVM which must, in some cases, "play OS" to adequately address progress and process control issues. The CDS2 API, a higher level of interface roughly equivalent in functionality to MPI and to be built entirely upon CDS1, is still being designed. It is intended to add support for the equivalent of communicators, reduction and other collective operations, process topologies, additional support for process creation, and some automatic memory management. CDS2 will not exactly match MPI, because the copy-free semantics of communication from CDS1 will be supported. CDS2 application programs will be free to carefully also use CDS1. CDS1 has been implemented on networks of workstations running unmodified Unix-based operating systems, using UDP/IP and vendor-supplied high- performance locks. Although its inter-node performance is currently unimpressive due to rudimentary implementation technique, it even now outperforms highly-optimized MPI implementation on intra-node communication due to its support for non-copy communication. The similarity of the CDS1 architecture to that of other projects such as UNET and TRAP suggests that the inter-node performance can be increased significantly to surpass MPI or PVM, and it may be possible to migrate some of its functionality to communication controllers.

DiNucci, David C.↗