Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Archive”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

ESS-DIVE guidelines for archiving terrestrial model data

This dataset contains supporting documents and images for ESS-DIVE terrestrial model data archiving guidelines.Terrestrial models are broadly defined as numerical models that couple both land dynamics and energy, water, carbon, or nutrient fluxes. We created these guidelines based on input from the U.S. Department of Energy’s Biological and Environmental Research land modeling community. The guidelines are intended to help modelers determine which components of their terrestrial model data associated with publication should be archived. Based on input from the land modeling community, the guidelines recommend archiving both model input and testing data, as well as code, script, and metadata. The guidelines also recommend archiving model data output, depending on the limitations set by data repositories. Lastly, we provide recommendations for bundling data files for publication as well as a discussion about tools that can facilitate model data archiving and reuse.This dataset is an archive of the associated GitHub repository for our model archiving guidelines (https://github.com/ess-dive-community/essdive-model-data-archiving-guidelines). The ‘README.pdf’ file gives a general introduction to the guidelines, and the ‘instructions.pdf’ file provides more detailed steps for following the guidelines. We also provide 2 figures in this data package: 1) a decision tree (model_data_guidelines_decision_tree.png) that can help users determine which components of their model data to archive. and 2) the ‘model_data_guidelines_flmd.png’ file depicts the different files that can be archived in addition to the model data itself. Lastly, we include 3 digitized tables from our associated manuscript and 3 CSV files with anonymized input from DOE scientists about the importance of different aspects of model data archiving from which we developed the guidelines.Dataset updates for v1.1.0: We updated this data package on 2021-11-22 in response to review comments on our related manuscript. In this update we removed one figure so that the model archiving guidelines are conveyed in text rather than an image. We updated the file-level metadata (FLMD) figure to be in accord with the most recent FLMD recommendations. We made minor edits to the README file to update the recommended citation and added two co-authors. We also added 6 new data files (3 are anonymized input from DOE scientists that helped to inform guidelines, and 3 are digitized tables from our manuscript.

54 ENVIRONMENTAL SCIENCES↗

The NAS Computational Aerosciences Archive

In order to further the state-of-the-art in computational aerosciences (CAS) technology, researchers must be able to gather and understand existing work in the field. One aspect of this information gathering is studying published work available in scientific journals and conference proceedings. However, current scientific publications are very limited in the type and amount of information that they can disseminate. Information is typically restricted to text, a few images, and a bibliography list. Additional information that might be useful to the researcher, such as additional visual results, referenced papers, and datasets, are not available. New forms of electronic publication, such as the World Wide Web (WWW), limit publication size only by available disk space and data transmission bandwidth, both of which are improving rapidly. The Numerical Aerodynamic Simulation (NAS) Systems Division at NASA Ames Research Center is in the process of creating an archive of CAS information on the WWW. This archive will be based on the large amount of information produced by researchers associated with the NAS facility. The archive will contain technical summaries and reports of research performed on NAS supercomputers, visual results (images, animations, visualization system scripts), datasets, and any other supporting meta-information. This information will be available via the WWW through the NAS homepage, located at http://www.nas.nasa.gov/, fully indexed for searching. The main components of the archive are technical summaries and reports, visual results, and datasets. Technical summaries are gathered every year by researchers who have been allotted resources on NAS supercomputers. These summaries, together with supporting visual results and references, are browsable by interested researchers. Referenced papers made available by researchers can be accessed through hypertext links. Technical reports are in-depth accounts of tools and applications research projects performed by NAS staff members and collaborators. Visual results, which may be available in the form of images, animations, and/or visualization scripts, are generated by researchers with respect to a certain research project, depicting dataset features that were determined important by the investigating researcher. For example, script files for visualization systems (e.g. FAST, PLOT3D, AVS) are provided to create visualizations on the user's local workstation to elucidate the key points of the numerical study. Users can then interact with the data starting where the investigator left off. Datasets are intended to give researchers an opportunity to understand previous work, 'mine' solutions for new information (for example, have you ever read a paper thinking "I wonder what the helicity density looks like?"), compare new techniques with older results, collaborate with remote colleagues, and perform validation. Supporting meta-information associated with the research projects is also important to provide additional context for research projects. This may include information such as the software used in the simulation (e.g. grid generators, flow solvers, visualization). In addition to serving the CAS research community, the information archive will also be helpful to students, visualization system developers and researchers, and management. Students (of any age) can use the data to study fluid dynamics, compare results from different flow solvers, learn about meshing techniques, etc., leading to better informed individuals. For these users it is particularly important that visualization be integrated into dataset archives. Visualization researchers can use dataset archives to test algorithms and techniques, leading to better visualization systems, Management can use the data to figure what is really going on behind the viewgraphs. All users will benefit from fast, easy, and convenient access to CFD datasets. The CAS information archive hopes to serve as a useful resource to those interested in computational sciences. At present, only information that may be distributed internationally is made available via the archive. Studies are underway to determine security requirements and solutions to make additional information available. By providing access to the archive via the WWW, the process of information gathering can be more productive and fruitful due to ease of access and ability to manage many different types of information. As the archive grows, additional resources from outside NAS will be added, providing a dynamic source of research results.

Miceli, Kristina D.↗

EOSDIS: Archive and Distribution Systems in the Year 2000

Earth Science Enterprise (ESE) is a long-term NASA research mission to study the processes leading to global climate change. The Earth Observing System (EOS) is a NASA campaign of satellite observatories that are a major component of ESE. The EOS Data and Information System (EOSDIS) is another component of ESE that will provide the Earth science community with easy, affordable, and reliable access to Earth science data. EOSDIS is a distributed system, with major facilities at seven Distributed Active Archive Centers (DAACs) located throughout the United States. The EOSDIS software architecture is being designed to receive, process, and archive several terabytes of science data on a daily basis. Thousands of science users and perhaps several hundred thousands of non-science users are expected to access the system. The first major set of data to be archived in the EOSDIS is from Landsat-7. Another EOS satellite, Terra, was launched on December 18, 1999. With the Terra launch, the EOSDIS will be required to support approximately one terabyte of data into and out of the archives per day. Since EOS is a multi-mission program, including the launch of more satellites and many other missions, the role of the archive systems becomes larger and more critical. In 1995, at the fourth convening of NASA Mass Storage Systems and Technologies Conference, the development plans for the EOSDIS information system and archive were described. Five years later, many changes have occurred in the effort to field an operational system. It is interesting to reflect on some of the changes driving the archive technology and system development for EOSDIS. This paper principally describes the Data Server subsystem including how the other subsystems access the archive, the nature of the data repository, and the mass-storage I/O management. The paper reviews the system architecture (both hardware and software) of the basic components of the archive. It discusses the operations concept, code development, and testing phase of the system. Finally, it describes the future plans for the archive.

Behnke, Jeanne↗

Integration experiences and performance studies of A COTS parallel archive systems

Current and future Archive Storage Systems have been asked to (a) scale to very high bandwidths, (b) scale in metadata performance, (c) support policy-based hierarchical storage management capability, (d) scale in supporting changing needs of very large data sets, (e) support standard interface, and (f) utilize commercial-off-the-shelf(COTS) hardware. Parallel file systems have been asked to do the same thing but at one or more orders of magnitude faster in performance. Archive systems continue to move closer to file systems in their design due to the need for speed and bandwidth, especially metadata searching speeds such as more caching and less robust semantics. Currently the number of extreme highly scalable parallel archive solutions is very small especially those that will move a single large striped parallel disk file onto many tapes in parallel. We believe that a hybrid storage approach of using COTS components and innovative software technology can bring new capabilities into a production environment for the HPC community much faster than the approach of creating and maintaining a complete end-to-end unique parallel archive software solution. In this paper, we relay our experience of integrating a global parallel file system and a standard backup/archive product with a very small amount of additional code to provide a scalable, parallel archive. Our solution has a high degree of overlap with current parallel archive products including (a) doing parallel movement to/from tape for a single large parallel file, (b) hierarchical storage management, (c) ILM features, (d) high volume (non-single parallel file) archives for backup/archive/content management, and (e) leveraging all free file movement tools in Linux such as copy, move, ls, tar, etc. We have successfully applied our working COTS Parallel Archive System to the current world's first petaflop/s computing system, LANL's Roadrunner, and demonstrated its capability to address requirements of future archival storage systems.

99 GENERAL AND MISCELLANEOUS↗

A Robust, Low-Cost Virtual Archive for Science Data

Despite their expense tape silos are still often the only affordable option for petabytescale science data archives, particularly when other factors such as data reliability, floor space, power and cooling load are accounted for. However, the complexity, management software, hardware reliability and access latency of tape silos make online data storage ever more attractive. Drastic reductions in low-cost mass-market PC disk drivers help to make this more affordable (approx. 1$/GB), but are challenging to scale to the petabyte range and of questionable reliability for archival use, On the other hand, if much of the science archive could be "virtualized", i.e., produced on demand when requested by users, we would need store only a fraction of the data online, perhaps bringing an online-only system into in affordable range. Radiance data from the satellite-borne Moderate Resolution Imaging Spectroradiometer (MODIS) instrument provides a good opportunity for such a virtual archive: the raw data amount to 140 GB/day, but these are small relative to the 550 GB/day making up the radiance products. These data are routinely processed as inputs for geophysical parameter products and then archived on tape at the Goddard Earth Sciences Distributed Active Archive (GES DAAC) for distributing to users. Virtualizing them would be an immediate and signifcant reduction in the amount of data being stored in the tape archives and provide more customizable products. A prototype of such a virtual archive is being developed to prove the concept and develop ways of incorporating the robustness that a science data archive requires.

Lynnes, Christopher↗

Integration experiments and performance studies of a COTS parallel archive system

Current and future Archive Storage Systems have been asked to (a) scale to very high bandwidths, (b) scale in metadata performance, (c) support policy-based hierarchical storage management capability, (d) scale in supporting changing needs of very large data sets, (e) support standard interface, and (f) utilize commercial-off-the-shelf (COTS) hardware. Parallel file systems have been asked to do the same thing but at one or more orders of magnitude faster in performance. Archive systems continue to move closer to file systems in their design due to the need for speed and bandwidth, especially metadata searching speeds such as more caching and less robust semantics. Currently the number of extreme highly scalable parallel archive solutions is very small especially those that will move a single large striped parallel disk file onto many tapes in parallel. We believe that a hybrid storage approach of using COTS components and innovative software technology can bring new capabilities into a production environment for the HPC community much faster than the approach of creating and maintaining a complete end-to-end unique parallel archive software solution. In this paper, we relay our experience of integrating a global parallel file system and a standard backup/archive product with a very small amount of additional code to provide a scalable, parallel archive. Our solution has a high degree of overlap with current parallel archive products including (a) doing parallel movement to/from tape for a single large parallel file, (b) hierarchical storage management, (c) ILM features, (d) high volume (non-single parallel file) archives for backup/archive/content management, and (e) leveraging all free file movement tools in Linux such as copy, move, Is, tar, etc. We have successfully applied our working COTS Parallel Archive System to the current world's first petafiop/s computing system, LANL's Roadrunner machine, and demonstrated its capability to address requirements of future archival storage systems.

99 GENERAL AND MISCELLANEOUS↗

Hashes are not suitable to verify fixity of the public archived web

Web archives, such as the Internet Archive, preserve the web and allow access to prior states of web pages. We implicitly trust their versions of archived pages, but as their role moves from preserving curios of the past to facilitating present day adjudication, we are concerned with verifying the fixity of archived web pages, or mementos, to ensure they have always remained unaltered. A widely used technique in digital preservation to verify the fixity of an archived resource is to periodically compute a cryptographic hash value on a resource and then compare it with a previous hash value. If the hash values generated on the same resource are identical, then the fixity of the resource is verified. We tested this process by conducting a study on 16,627 mementos from 17 public web archives. We replayed and downloaded the mementos 39 times using a headless browser over a period of 442 days and generated a hash for each memento after each download, resulting in 39 hashes per memento. The hash is calculated by including not only the content of the base HTML of a memento but also all embedded resources, such as images and style sheets. We expected to always observe the same hash for a memento regardless of the number of downloads. However, our results indicate that 88.45% of mementos produce more than one unique hash value, and about 16% (or one in six) of those mementos always produce different hash values. We identify and quantify the types of changes that cause the same memento to produce different hashes. These results point to the need for defining an archive-aware hashing function, as conventional hashing functions are not suitable for replayed archived web pages.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

WE-H-204-02: Part 1: History and Archives Resources at AIP for AAPM and Its Members

“William D. Coolidge, Inventor of the Modern X-ray Tube” David J. Allard, M.S., CHP - Director, PA DEP Bureau of Radiation Protection William David Coolidge 1873–1975 was a research scientist and inventor of the modern X-ray tube. Besides Roentgen, with his 1895 discovery and subsequent studies of X-rays, perhaps no other individual contributed more to the advancement of X-ray technology than did Coolidge. He was born in Hudson, MA and received his Bachelor of Science degree from MIT in 1896. That same year he went to Europe to study under renowned physicists of the time. Coolidge received his Ph.D. summa cum laude from the University of Leipzig in 1899 and soon after joined the staff of MIT. While studying at Leipzig, he met Roentgen. In 1905 he was asked to join the newly established General Electric Research Laboratory in Schenectady, NY. He promptly began fundamental work on the production of ductile tungsten filaments as a replacement for fragile carbon filaments used in incandescent light bulbs. This improved light bulb was brought to market by GE in 1911. It was subsequent application of his tungsten work that led Coolidge to his studies in X ray production. Circa 1910, the state-of-the-art X-ray tube was a “gas tube” or “cold cathode” type tube. These crude X-ray tubes relied on residual gas molecules as a source of electrons for bombardment of low to medium atomic number metal targets. In 1912 Coolidge described the use of tungsten as an improved anode target material for X-ray tubes. Shortly after in 1913 he published a paper in Physical Review describing “A Powerful Roentgen Ray Tube With a Pure Electron Discharge.” This tube used a tungsten filament as a thermionic source of electrons under high vacuum to bombard a tungsten anode target. Great improvements in X-ray tube stability, output and performance were obtained with the “hot cathode” or “Coolidge tube.” With some variation in filament and target geometry, this 100 year old invention is the same basic X-ray tube used today in medicine, research and industry. In 1932 Coolidge became Director of the GE Laboratory, then in 1940 Vice-President and Director of Research. In 1941 he was a member of a small committee, appointed by President Franklin D. Roosevelt, to evaluate the military importance of research on uranium. This committee’s report led to the establishment of the Manhattan Engineering District for nuclear weapons development during WWII. Coolidge lived to be over 100 years old, he had 83 patents to his credit, numerous awards and honorary degrees, and in 1975 was elected to the National Inventor’s Hall of Fame. At the time he was the only inventor to receive this honor in his lifetime. Dr. Coolidge was also the first recipient of the AAPM’s highest science award - named in his honor. From notes of a day-long interview with Coolidge’s son Lawrence in the mid-1990s, previous biographies, publications, books, GE literature, historic photographs, e.g., a wonderful 1874 photo stereoview card with 1 year old baby “Willie Coolidge”, and other artifacts in the author’s collection, this presentation will review Dr. Coolidge’s amazing life, work, accomplishments and awards. “History and Archives Resources at AIP for AAPM and its Members” Gregory A. Good, Ph.D. - Director, AIP Center for History of Physics Melanie J. Mueller, MLIS - Acting Director, AIP Niels Bohr Library & Archives The American Institute of Physics established the Center for History of Physics and the Niels Bohr Library & Archives in the 1960s. Our shared mission is: To preserve and make known the history of the physical sciences. This talk will explore the many ways that AIP’s two history programs support the historical and archival activities of AAPM. Topics will include our ongoing oral history program, web outreach through exhibits and teaching guides, and archiving for AAPM and other Member Societies. We will focus in particular on materials in our collections related to the history of medical physics and to the history of AAPM. We will unveil and demonstrate a new “Archives Portal” that we are designing specifically to be useful to AAPM and its members. Learning Objectives: Study the background of the medical physicist - William David Coolidge Examine the time-line for his success Review the publications conceptualizing his works and progressions Realize what he invented Evaluate the importance of the invention Relate the success to national prominence Uncover how he influenced medical physicists today Find out how he was celebrated by the AAPM View the AIP established Center for History of Physics Consider the significant efforts and vision to preserve the history of medical physics Learn about the Niels Bohr Library & Archives Look back in time at medical physics in the 1960s Unveil and demonstrate a new “Archives Portal” that will be useful to AAPM.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

WE-H-204-03: Part 2: History and Archives Resources at AIP for AAPM and Its Members

“William D. Coolidge, Inventor of the Modern X-ray Tube” David J. Allard, M.S., CHP - Director, PA DEP Bureau of Radiation Protection William David Coolidge 1873–1975 was a research scientist and inventor of the modern X-ray tube. Besides Roentgen, with his 1895 discovery and subsequent studies of X-rays, perhaps no other individual contributed more to the advancement of X-ray technology than did Coolidge. He was born in Hudson, MA and received his Bachelor of Science degree from MIT in 1896. That same year he went to Europe to study under renowned physicists of the time. Coolidge received his Ph.D. summa cum laude from the University of Leipzig in 1899 and soon after joined the staff of MIT. While studying at Leipzig, he met Roentgen. In 1905 he was asked to join the newly established General Electric Research Laboratory in Schenectady, NY. He promptly began fundamental work on the production of ductile tungsten filaments as a replacement for fragile carbon filaments used in incandescent light bulbs. This improved light bulb was brought to market by GE in 1911. It was subsequent application of his tungsten work that led Coolidge to his studies in X ray production. Circa 1910, the state-of-the-art X-ray tube was a “gas tube” or “cold cathode” type tube. These crude X-ray tubes relied on residual gas molecules as a source of electrons for bombardment of low to medium atomic number metal targets. In 1912 Coolidge described the use of tungsten as an improved anode target material for X-ray tubes. Shortly after in 1913 he published a paper in Physical Review describing “A Powerful Roentgen Ray Tube With a Pure Electron Discharge.” This tube used a tungsten filament as a thermionic source of electrons under high vacuum to bombard a tungsten anode target. Great improvements in X-ray tube stability, output and performance were obtained with the “hot cathode” or “Coolidge tube.” With some variation in filament and target geometry, this 100 year old invention is the same basic X-ray tube used today in medicine, research and industry. In 1932 Coolidge became Director of the GE Laboratory, then in 1940 Vice-President and Director of Research. In 1941 he was a member of a small committee, appointed by President Franklin D. Roosevelt, to evaluate the military importance of research on uranium. This committee’s report led to the establishment of the Manhattan Engineering District for nuclear weapons development during WWII. Coolidge lived to be over 100 years old, he had 83 patents to his credit, numerous awards and honorary degrees, and in 1975 was elected to the National Inventor’s Hall of Fame. At the time he was the only inventor to receive this honor in his lifetime. Dr. Coolidge was also the first recipient of the AAPM’s highest science award - named in his honor. From notes of a day-long interview with Coolidge’s son Lawrence in the mid-1990s, previous biographies, publications, books, GE literature, historic photographs, e.g., a wonderful 1874 photo stereoview card with 1 year old baby “Willie Coolidge”, and other artifacts in the author’s collection, this presentation will review Dr. Coolidge’s amazing life, work, accomplishments and awards. “History and Archives Resources at AIP for AAPM and its Members” Gregory A. Good, Ph.D. - Director, AIP Center for History of Physics Melanie J. Mueller, MLIS - Acting Director, AIP Niels Bohr Library & Archives The American Institute of Physics established the Center for History of Physics and the Niels Bohr Library & Archives in the 1960s. Our shared mission is: To preserve and make known the history of the physical sciences. This talk will explore the many ways that AIP’s two history programs support the historical and archival activities of AAPM. Topics will include our ongoing oral history program, web outreach through exhibits and teaching guides, and archiving for AAPM and other Member Societies. We will focus in particular on materials in our collections related to the history of medical physics and to the history of AAPM. We will unveil and demonstrate a new “Archives Portal” that we are designing specifically to be useful to AAPM and its members. Learning Objectives: Study the background of the medical physicist - William David Coolidge Examine the time-line for his success Review the publications conceptualizing his works and progressions Realize what he invented Evaluate the importance of the invention Relate the success to national prominence Uncover how he influenced medical physicists today Find out how he was celebrated by the AAPM View the AIP established Center for History of Physics Consider the significant efforts and vision to preserve the history of medical physics Learn about the Niels Bohr Library & Archives Look back in time at medical physics in the 1960s Unveil and demonstrate a new “Archives Portal” that will be useful to AAPM.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Interim report on Landsat national archive activities

The Department of the Interior (DOI) has the responsibility to preserve and to distribute most Landsat Thematic Mapper (TM) and Multispectral Scanner (MSS) data that have been acquired by the five Landsat satellites operational since July 1972. Data that are still covered by exclusive marketing rights, which were granted by the U.S. Government to the commercial Landsat operator, cannot be distributed by the DOI. As the designated national archive for Landsat data, the U.S. Geological Survey's EROS Data Center (EDC) has initiated two new programs to protect and make available any of the 625,000 MSS scenes currently archived and the 200,000 TM scenes to be archived at EDC by 1995. A specially configured system has begun converting Landsat MSS data from obsolete high density tapes (HDT's) to more dense digital cassette tapes. After transcription, continuous satellite swaths are (1) divided into standard scenes defined by a world reference system, (2) geographically located by latitude and longitude, and (3) assessed for overall quality. Digital browse images are created by subsampling the full-resolution swaths. Conversion of the TM HDT's will begin in the fourth quarter of 1992 and will be conducted concurrently with MSS conversion. Although the TM archive is three times larger than the entire MSS archive, conversion of data from both sensor systems and consolidation of the entire Landsat archive at EDC will be completed by the end of 1994. Some MSS HDT's have deteriorated, primarily as a result of hydrolysis of the pigment binder. Based on a small sample of the 11 terabytes of post-1978 MSS data and the 41 terabytes of TM data to be converted, it appears that to date, less than 2 percent of the data have been lost. The data loss occurs within small portions of some scenes; few scenes are lost entirely. Approximately 10,000 pre-1979 MSS HDT's have deteriorated to such an extent, as a result of hydrolysis, that the data cannot be recovered without special treatment of the tapes. An independent consulting division of a major tape manufacturer has analyzed affected tapes and is confident that restorative procedures can be applied to the HDT's to permit one pass to reproduce the data on another recording media. A system to distribute minimally processed Landsat data will be procured in 1992 and will be operational by mid-1994. Any TM or MSS data in the national archive that are not restricted by exclusive marketing rights will be reproduced directly from the archive media onto user specified computer-compatible media. TM data will be produced either at a raw level (radiometrically and geometrically uncorrected) or at an intermediate level (radiometrically corrected and geometrically indexed). MSS data will be produced to an intermediate level or to a fully corrected level (radiometrically corrected and geometrically transformed to an Oblique Mercator projection). The system will be capable of providing ordered scenes within 48 hours of receipt of order.

Boyd, John E.↗

Archiving Microgravity Flight Data and Samples

To obtain help in evaluating its current strategy for archiving data and samples obtained in microgravity research, NASA's Microgravity Science and Applications Division (MSAD) asked the Space Studies Board's Committee on Microgravity Research for guidance on the following questions: What data should be archived and where should it be kept? In what form should the data be maintained (electronic files, photographs, hard copy, samples)? What should the general format of the database be? To what extent should it be universally accessible and through what mechanisms? Should there be a period of time for which principal investigators have proprietary access? If so, how long should proprietary data be stored? What provisions should be made for data obtained from ground-based experiments? What should the deadline be for investigators placing their data in the archive? How long should data be saved? How long should data be easily accessible? As a prelude to making recommendations for optimum selection and storage of microgravity data and samples, the committee in this report briefly describes NASA's past archiving practices and outlines MSAD's current archiving strategy. Although the committee found that only a limited number of experiments have thus far been archived, it concluded that the general archiving strategy, characterized by MSAD as minimalist, appears viable. A central focus of attention is the Experiment Data Management Plan (EDMP), MSAD's recently instituted data management and archiving framework for flight experiments. Many of the report's recommendations are aimed at enhancing the effectiveness of the EDMP approach, which the committee regards as an appropriate data management method for MSAD. Other recommendations provide guidance on broader issues related to the questions listed above. This report does not address statutory or regulatory records retention requirements.

Source record↗

A MySQL Based EPICS Archiver

Archiving a large fraction of the EPICS signals within the Jefferson Lab (JLAB) Accelerator control system is vital for postmortem and real-time analysis of the accelerator performance. This analysis is performed on a daily basis by scientists, operators, engineers, technicians, and software developers. Archiving poses unique challenges due to the magnitude of the control system. A MySQL Archiving system (Mya) was developed to scale to the needs of the control system; currently archiving 58,000 EPICS variables, updating at a rate of 11,000 events per second. In addition to the large collection rate, retrieval of the archived data must also be fast and robust. Archived data retrieval clients obtain data at a rate over 100,000 data points per second. Managing the data in a relational database provides a number of benefits. This paper describes an archiving solution that uses an open source database and standard off the shelf hardware to reach high performance archiving needs. Mya has been in production at Jefferson Lab since February of 2007.

43 PARTICLE ACCELERATORS↗

Long-term data archiving

Long term data archiving has much value for chemists, not only to retain access to research and product development records, but also to enable new developments and new discoveries. There are some recent regulatory requirements (e.g., FDA 21 CFR Part 11), but good science and good business both benefit regardless. A particular example of the benefits of and need for long term data archiving is the management of data from spectroscopic laboratory instruments. The sheer amount of spectroscopic data is increasing at a scary rate, and the pressures to archive come from the expense to create the data (or recreate it if it is lost) as well as its high information content. The goal of long-term data archiving is to save and organize instrument data files as well as any needed meta data (such as sample ID, LIMS information, operator, date, time, instrument conditions, sample type, excitation details, environmental parameters, etc.). This editorial explores the issues involved in long-term data archiving using the example of Raman spectral databases. There are at present several such databases, including common data format libraries and proprietary libraries. However, such databases and libraries should ultimately satisfy stringent criteria for long term data archiving, including readability for long times into the future, robustness to changes in computer hardware and operating systems, and use of public domain data formats. The latter criterion implies the data format should be platform independent and the tools to create the data format should be easily and publicly obtainable or developable. Several examples of attempts at spectral libraries exist, such as the ASTM ANDI format, and the JCAMP-DX format. On the other hand, proprietary library spectra can be exchanged and manipulated using proprietary tools. As the above examples have deficiencies according to the three long term data archiving criteria, Extensible Markup Language (XML; a product of the World Wide Web Consortium, an independent standards body) as a new data interchange tool is being investigated and implemented. In order to facilitate data archiving, Raman data needs calibration as well as some other kinds of data treatment. Figure 1 illustrates schematically the present situation for Raman data calibration in the world-wide Raman spectroscopy community, and presents some of the terminology used.

99 GENERAL AND MISCELLANEOUS↗

Current status of the international Halley Watch infrared net archive

The primary purposes of the Halley Watch have been to promote Halley observations, coordinate and standardize the observing where useful, and to archive the results in a database readily accessible to cometary scientists. The intention of IHW is to store the observations themselves, along with any information necessary to allow users to understand and use the data, but to exclude interpretations of these data. Each of the archives produced by the IHW will appear in two versions: a printed archive and a digital archive on CD-ROMs. The archive is expected to have a very long lifetime. The IHW has already produced an archive for P/Crommelin. This consists of one printed volume and two 1600 bpi tapes. The Halley archive will contain at least twenty gigabytes of information.

Mcguinness, Brian B.↗

Stewardship of very large digital data archives

An archive is a permanent store. There are relatively few very large digital data archives in existence. Most business records are expired within five or ten years. Many kinds of business records that do have long lives are embedded in data bases that are continually updated and re-issued cyclically. Also, a great deal of permanent business records are actually archived as microfilm, fiche, or optical disk images - their digital version being an operational convenience rather than an archive. The problems forseen in stewarding the very large digital data archives that will accumulate during the mission of the Earth Observing System (EOS) are addressed. It focuses on the function of shepherding archived digital data into an endless future. Stewardship entails storing and protecting the archive and providing meaningful service to the community of users. The steward will (1) provide against loss due to physical phenomena; (2) assure that data is not lost due to storage technology obsolescence; and (3) maintain data in a current formatting methodology.

Savage, Patric↗

Recommendations for a service framework to access astronomical archives

There are a large number of astronomical archives and catalogs on-line for network access, with many different user interfaces and features. Some systems are moving towards distributed access, supplying users with client software for their home sites which connects to servers at the archive site. Many of the issues involved in defining a standard framework of services that archive/catalog suppliers can use to achieve a basic level of interoperability are described. Such a framework would simplify the development of client and server programs to access the wide variety of astronomical archive systems. The primary services that are supplied by current systems include: catalog browsing, dataset retrieval, name resolution, and data analysis. The following issues (and probably more) need to be considered in establishing a standard set of client/server interfaces and protocols: Archive Access - dataset retrieval, delivery, file formats, data browsing, analysis, etc.; Catalog Access - database management systems, query languages, data formats, synchronous/asynchronous mode of operation, etc.; Interoperability - transaction/message protocols, distributed processing mechanisms (DCE, ONC/SunRPC, etc), networking protocols, etc.; Security - user registration, authorization/authentication mechanisms, etc.; Service Directory - service registration, lookup, port/task mapping, parameters, etc.; Software - public vs proprietary, client/server software, standard interfaces to client/server functions, software distribution, operating system portability, data portability, etc. Several archive/catalog groups, notably the Astrophysics Data System (ADS), are already working in many of these areas. In the process of developing StarView, which is the user interface to the Space Telescope Data Archive and Distribution Service (ST-DADS), these issues and the work of others were analyzed. A framework of standard interfaces for accessing services on any archive system which would benefit archive user and supplier alike is proposed.

Travisano, J. J.↗

Architecture and evolution of Goddard Space Flight Center Distributed Active Archive Center

The Goddard Space Flight Center (GSFC) Distributed Active Archive Center (DAAC) has been developed to enhance Earth Science research by improved access to remote sensor earth science data. Building and operating an archive, even one of a moderate size (a few Terabytes), is a challenging task. One of the critical components of this system is Unitree, the Hierarchical File Storage Management System. Unitree, selected two years ago as the best available solution, requires constant system administrative support. It is not always suitable as an archive and distribution data center, and has moderate performance. The Data Archive and Distribution System (DADS) software developed to monitor, manage, and automate the ingestion, archive, and distribution functions turned out to be more challenging than anticipated. Having the software and tools is not sufficient to succeed. Human interaction within the system must be fully understood to improve efficiency to improve efficiency and ensure that the right tools are developed. One of the lessons learned is that the operability, reliability, and performance aspects should be thoroughly addressed in the initial design. However, the GSFC DAAC has demonstrated that it is capable of distributing over 40 GB per day. A backup system to archive a second copy of all data ingested is under development. This backup system will be used not only for disaster recovery but will also replace the main archive when it is unavailable during maintenance or hardware replacement. The GSFC DAAC has put a strong emphasis on quality at all level of its organization. A Quality team has also been formed to identify quality issues and to propose improvements. The DAAC has conducted numerous tests to benchmark the performance of the system. These tests proved to be extremely useful in identifying bottlenecks and deficiencies in operational procedures.

Bedet, Jean-Jacques↗

HEASARC - The High Energy Astrophysics Science Archive Research Center

The High Energy Astrophysics Science Archive Research Center (HEASARC) is NASA's archive for high-energy astrophysics and cosmic microwave background (CMB) data, supporting the broad science goals of NASA's Physics of the Cosmos theme. It provides vital scientific infrastructure to the community by standardizing science data formats and analysis programs, providing open access to NASA resources, and implementing powerful archive interfaces. Over the next five years the HEASARC will ingest observations from up to 12 operating missions, while serving data from these and over 30 archival missions to the community. The HEASARC archive presently contains over 37 TB of data, and will contain over 60 TB by the end of 2014. The HEASARC continues to secure major cost savings for NASA missions, providing a reusable mission-independent framework for reducing, analyzing, and archiving data. This approach was recognized in the NRC Portals to the Universe report (2007) as one of the HEASARC's great strengths. This poster describes the past and current activities of the HEASARC and our anticipated developments in coming years. These include preparations to support upcoming high energy missions (NuSTAR, Astro-H, GEMS) and ground-based and sub-orbital CMB experiments, as well as continued support of missions currently operating (Chandra, Fermi, RXTE, Suzaku, Swift, XMM-Newton and INTEGRAL). In 2012 the HEASARC (which now includes LAMBDA) will support the final nine-year WMAP data release. The HEASARC is also upgrading its archive querying and retrieval software with the new Xamin system in early release - and building on opportunities afforded by the growth of the Virtual Observatory and recent developments in virtual environments and cloud computing.

Smale, Alan P.↗