Results for “file system” — Page 58

NASA NTRSDate not supplied

Simplifying Analysis of Hierarchical HDF5 and NetCDF4 Files with Xarray-Datatree

NASA’s Earth Observing System Data and Information System (EOSDIS) contains thousands of Earth science datasets from satellites, models, and field campaigns. EOSDIS data are stored in formats that are well supported by the Earth Science community. These formats include the Hierarchical Data Format (HDF), with derivative flavors such as HDF-5 and the Network Common Data Format (NetCDF-4). The HDF specification allows for a directory-like hierarchy within a single file, known as "groups". Observational data and associated metadata within a single file can be distributed amongst multiple internal groups, which can also be nested to multiple levels. Working with datasets that have a group hierarchical structure can be difficult because of the nested structure of groups. Widely used packages, such as xarray, have data models that do not accommodate the hierarchical structure within HDF files, requiring users to traverse the file and open different HDF groups as separate, unrelated objects. Xarray-datatree is a Python package developed to solve the difficulty of traversing HDFs with a hierarchical group structure by creating a tree-like hierarchical data structure in xarray. The tree-like structure allows each group to be accessed once a DataTree object is instantiated. The migration of xarray-datatree into the xarray core library will reduce barriers to accessing Earth science data by eliminating the need to understand and traverse the specific hierarchy of a grouped HDF file.

Eni Awowale↗

DOE OSTI2017

File format for normalizing radiological concentration exposure rate and dose rate data for the effects of radioactive decay and weathering processes

This report specifies the electronic file format that was agreed upon to be used as the file format for normalized radiological data produced by the software tool developed under this TI project. The NA-84 Technology Integration (TI) Program project (SNL17-CM-635, Normalizing Radiological Data for Analysis and Integration into Models) investigators held a teleconference on December 7, 2017 to discuss the tasks to be completed under the TI program project. During this teleconference, the TI project investigators determined that the comma-separated values (CSV) file format is the most suitable file format for the normalized radiological data that will be outputted from the normalizing tool developed under this TI project. The CSV file format was selected because it provides the requisite flexibility to manage different types of radiological data (i.e., activity concentration, exposure rate, dose rate) from other sources [e.g., Radiological Assessment and Monitoring System (RAMS), Aerial Measuring System (AMS), Monitoring and Sampling). The CSV file format also is suitable for the file format of the normalized radiological data because this normalized data can then be ingested by other software [e.g., RAMS, Visual Sampling Plan (VSP)] used by the NA-84’s Consequence Management Program.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2016

A Radiation-Triggered Surveillance System for UF6 Cylinder Monitoring

This report provides background information and representative scenarios for testing a prototype radiation-triggered surveillance system at an operating facility that handles uranium hexafluoride (UF 6 ) cylinders. The safeguards objective is to trigger cameras using radiation, or radiation and motion, rather than motion alone, to reduce significantly the number of image files generated by a motion-triggered system. The authors recommend the use of radiation-triggered surveillance at all facilities where cylinder paths are heavily traversed by personnel. The International Atomic Energy Agency (IAEA) has begun using surveillance cameras in the feed and withdrawal areas of gas centrifuge enrichment plants (GCEPs). The cameras generate imagery using elapsed time or motion, but this creates problems in areas occupied 24/7 by personnel. Either motion-or-interval-based triggering generates thousands of review files over the course of a month. Since inspectors must review the files to verify operator material-flow-declarations, a plethora of files significantly extends the review process. The primary advantage of radiation-triggered surveillance is the opportunity to obtain full-time cylinder throughput verification versus what presently amounts to part-time verification. Cost savings should be substantial, as the IAEA presently uses frequent unannounced inspections to verify cylinder-throughput declarations. The use of radiation-triggered surveillance allows the IAEA to implement less frequent unannounced inspections for the purpose of flow verification, but its principal advantage is significantly shorter and more effective inspector video reviews.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

NASA NTRS2008

Aquarius's Instrument Science Data System (ISDS) Automated to Acquire, Process, Trend Data and Produce Radiometric System Assessment Reports

The Aquarius Radiometer, a subsystem of the Aquarius Instrument required a data acquisition ground system to support calibration and radiometer performance assessment. To support calibration and compose performance assessments, we developed an automated system which uploaded raw data to a ftp server and saved raw and processed data to a database. This paper details the overall functionalities of the Aquarius Instrument Science Data System (ISDS) and the individual electrical ground support equipment (EGSE) which produced data files that were infused into the ISDS. Real time EGSEs include an ICDS Simulator, Calibration GSE, Labview controlled power supply, and a chamber data acquisition system. ICDS Simulator serves as a test conductor primary workstation, collecting radiometer housekeeping (HK) and science data and passing commands and HK telemetry collection request to the radiometer. Calibration GSE (Radiometer Active Test Source) provides source choice from multiple targets for the radiometer external calibration. Power Supply GSE, controlled by labview, provides real time voltage and current monitoring of the radiometer. And finally the chamber data acquisition system produces data reflecting chamber vacuum pressure, thermistor temperatures, AVG and watts. Each GSE system produce text based data files every two to six minutes and automatically copies the data files to the Central Archiver PC. The Archiver PC stores the data files, schedules automated uploads of these files to an external FTP server, and accepts request to copy all data files to the ISDS for offline data processing and analysis. Aquarius Radiometer ISDS contains PHP and MATLab programs to parse, process and save all data to a MySQL database. Analysis tools (MATLab programs) in the ISDS system are capable of displaying radiometer science, telemetry and auxiliary data in near real time as well as performing data analysis and producing automated performance assessment reports of the Aquarius Radiometer.

Source record↗

DOE OSTI2018

Data Import Manager

The Data Import Manager is a part of PRISM, the Proteomics Research Information and Management System. The Data Import Manager monitors a central folder for new dataset trigger files, which are XML files that specify metadata for new datasets to import into the Data Management System (DMS). The Data Import Manager validates the metadata in the XML trigger file, notifying the instrument operator by e-mail if invalid metadata is encountered. After processing each XML trigger file, the file is moved into either a success or failure folder.

Monroe, Matthew↗

DOE OSTI2014

MO-A-BRD-10: A Fast and Accurate GPU-Based Proton Transport Monte Carlo Simulation for Validating Proton Therapy Treatment Plans

Purpose: To build a GPU-based Monte Carlo (MC) simulation of proton transport with detailed modeling of elastic and non-elastic (NE) protonnucleus interactions, for use in a very fast and cost-effective proton therapy treatment plan verification system. Methods: Using the CUDA framework, we implemented kernels for the following tasks: (1) Simulation of beam spots from our possible scanning nozzle configurations, (2) Proton propagation through CT geometry, taking into account nuclear elastic and multiple scattering, as well as energy straggling, (3) Bertini-style modeling of the intranuclear cascade stage of NE interactions, and (4) Simulation of nuclear evaporation. To validate our MC, we performed: (1) Secondary particle yield calculations in NE collisions with therapeutically-relevant nuclei, (2) Pencil-beam dose calculations in homogeneous phantoms, (3) A large number of treatment plan dose recalculations, and compared with Geant4.9.6p2/TOPAS. A workflow was devised for calculating plans from a commercially available treatment planning system, with scripts for reading DICOM files and generating inputs for our MC. Results: Yields, energy and angular distributions of secondaries from NE collisions on various nuclei are in good agreement with the Geant4.9.6p2 Bertini and Binary cascade models. The 3D-gamma pass rate at 2%–2mm for 70–230 MeV pencil-beam dose distributions in water, soft tissue, bone and Ti phantoms is 100%. The pass rate at 2%–2mm for treatment plan calculations is typically above 98%. The net computational time on a NVIDIA GTX680 card, including all CPU-GPU data transfers, is around 20s for 1×10{sup 7} proton histories. Conclusion: Our GPU-based proton transport MC is the first of its kind to include a detailed nuclear model to handle NE interactions on any nucleus. Dosimetric calculations demonstrate very good agreement with Geant4.9.6p2/TOPAS. Our MC is being integrated into a framework to perform fast routine clinical QA of pencil-beam treatment plans. Hardware for such a system will cost under $5000. This work was funded in part by a grant from Varian Medical Systems, Inc.

61 RADIATION PROTECTION AND DOSIMETRY↗

NASA NTRS1989

PARAGRAPH: A graphics tool for performance and reliability analysis

PARAGRAPH is an animated graphics display package. It consists of two parts: an interface to CSIM (a process based simulation language) and a graphic display system. When a simulation model is executed on CSIM the interface collects pertinent performance analysis data and writes them to a file. This file is then fed to the graphic display system which depicts the execution of the simulation model visually. This report focuses on the graphical display system. Specifically it describes the user interface and features of the display system. It also explains how Interviews (which is based on X-windows) is used as the basis for the design of PARAGRAPH. The last section contains an example in which a basic load balancing simulation model is used to demonstrate the features and the capability of PARAGRAPH.

Lee, Kevin Douglas↗

DOE OSTI2012

Laboratory Guide for Residual Stress Sample Alignment and Experiment Planning-October 2011 Version

The December 2010 version of the guide, ORNL/TM-2008/159, by Jeff Bunn, Josh Schmidlin, Camden Hubbard, and Paris Cornwell, has been further revised due to a major change in the GeoMagic Studio software for constructing a surface model. The Studio software update also includes a plug-in module to operate the FARO Scan Arm. Other revisions for clarity were also made. The purpose of this revision document is to guide the reader through the process of laser alignment used by NRSF2 at HFIR and VULCAN at SNS. This system was created to increase the spatial accuracy of the measurement points in a sample, reduce the use of neutron time used for alignment, improve experiment planning, and reduce operator error. The need for spatial resolution has been driven by the reduction in gauge volumes to the sub-millimeter level, steep strain gradients in some samples, and requests to mount multiple samples within a few days for relating data from each sample to a common sample coordinate system. The first step in this process involves mounting the sample on an indexer table in a laboratory set up for offline sample mounting and alignment in the same manner it would be mounted at either instrument. In the shared laboratory, a FARO ScanArm is used to measure the coordinates of points on the sample surface ('point cloud'), specific features and fiducial points. A Sample Coordinate System (SCS) needs to be established first. This is an advantage of the technique because the SCS can be defined in such a way to facilitate simple definition of measurement points within the sample. Next, samples are typically mounted to a frame of 80/20 and fiducial points are attached to the sample or frame then measured in the established sample coordinate system. The laser scan probe on the ScanArm can then be used to scan in an 'as-is' model of the sample as well as mounting hardware. GeoMagic Studio 12 is the software package used to construct the model from the point cloud the scan arm creates. Once a model, fiducial, and measurement files are created, a special program, called SScanSS combines the information and by simulation of the sample on the diffractometer can help plan the experiment before using neutron time. Finally, the sample is mounted on the relevant stress measurement instrument and the fiducial points are measured again. In the HFIR beam room, a laser tracker is used in conjunction with a program called CAM2 to measure the fiducial points in the NRSF2 instrument's sample positioner coordinate system. SScanSS is then used again to perform a coordinate system transformation of the measurement file locations to the sample positioner coordinate system. A procedure file is then written with the coordinates in the sample positioner coordinate system for the desired measurement locations. This file is often called a script or command file and can be further modified using excel. It is very important to note that this process is not a linear one, but rather, it often is iterative. Many of the steps in this guide are interdependent on one another. It is very important to discuss the process as it pertains to the specific sample being measured. What works with one sample may not necessarily work for another. This guide attempts to provide a typical work flow that has been successful in most cases.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA NTRSDate not supplied

Acquisition of and Access to Research Omics Data

Omics data are essential for understanding the myriad and complex effects of space environments on humans. To assure maximum benefit from these kinds of data, the NASA Human Research Program Data Management Plan stipulates that human omics data should be archived within and accessed through the NASA Life Sciences Portal (NLSP). The NLSP has the capability to acquire and provision access to omics (and other kinds of) research results for individual and ad-hoc groups of subjects at the direction of institutional review boards, or other authorizing bodies or individuals, per institutional, program and investigation-specific policies and procedures. However, because some single-subject omics data, like CT scans and other kinds of large, complex biomedical data, could be used to identify heretofore unknown risks to the subject’s health, or, in certain cases, be used to identify a subject, NASA Policy Directive 7170.1 describes various policies regarding the management of and access to “research genetic testing” data, which includes many kinds of omics data. For example, NPD 7170.1 prohibits access to human research genetic data by NASA personnel who make employment decisions for the subjects from whom the data were obtained. To meet the objective of acquiring research omics data for NLSP in compliance with the policies in NPD 7170.1 and other applicable NASA policies, we designed NOMADS (the NLSP Omics Multimodal Acquisition of Data System), a new component that supports the transfer of large research data files, including research genetic testing data, using one of several different transfer mechanisms. The choice of mechanism is made by the submitter of the data, with guiding information from the system, and is likely to often be determined in large part by the nature and source location of the data. For example, for small files where the source data files are not already stored in a cloud storage system, users are likely to prefer to transfer their data to the NLSP via a web browser. Conversely, for large sets of files already organized and stored in a cloud storage system, users may opt for NOMAD’s cloud-to-cloud transfer method. All omics datasets targeted for the NASA Life Sciences Data Archive must pass a variety of quality checks to ensure data integrity and adherence to the standards defined by the LSDA Data Submission Guidelines (DSG) (see https://nlsp.nasa.gov/explore/lsdahome/datasubmit). These include requirements that data are consistent with open standards established by the omics community. Non-compliant data will not be accepted however archivists are available to advise submitters on how to revise data submissions and re-submit until compliance is achieved. Following compliance with the LSDA DSG, omics data next undergo a variety of additional quality checks to ensure the data meet omics community standards. Domain specific Omics data quality control tools and techniques are continually evolving and linked to the advancements in omics assays utilized and thus, the tools and techniques utilized by the LSDA for data quality control and validation will need to be sustained accordingly. All human omics data will be access controlled according to the policies described above, and requiring IRB approval for any additional access grants once the data are acquired (including access for analysis using the NLSP workspace tools).

Omics↗

NASA NTRS2024

Acquisition of and Access to Research Omics Data

Omics data are essential for understanding the myriad and complex effects of space environments on humans. To assure maximum benefit from these kinds of data, the NASA Human Research Program Data Management Plan stipulates that human omics data should be archived within and accessed through the NASA Life Sciences Portal (NLSP). The NLSP has the capability to acquire and provision access to omics (and other kinds of) research results for individual and ad-hoc groups of subjects at the direction of institutional review boards, or other authorizing bodies or individuals, per institutional, program and investigation-specific policies and procedures. However, because some single-subject omics data, like CT scans and other kinds of large, complex biomedical data, could be used to identify heretofore unknown risks to the subject’s health, or, in certain cases, be used to identify a subject, NASA Policy Directive 7170.1 describes various policies regarding the management of and access to “research genetic testing” data, which includes many kinds of omics data. For example, NPD 7170.1 prohibits access to human research genetic data by NASA personnel who make employment decisions for the subjects from whom the data were obtained. To meet the objective of acquiring research omics data for NLSP in compliance with the policies in NPD 7170.1 and other applicable NASA policies, we designed NOMADS (the NLSP Omics Multimodal Acquisition of Data System), a new component that supports the transfer of large research data files, including research genetic testing data, using one of several different transfer mechanisms. The choice of mechanism is made by the submitter of the data, with guiding information from the system, and is likely to often be determined in large part by the nature and source location of the data. For example, for small files where the source data files are not already stored in a cloud storage system, users are likely to prefer to transfer their data to the NLSP via a web browser. Conversely, for large sets of files already organized and stored in a cloud storage system, users may opt for NOMAD’s cloud-to-cloud transfer method. All omics datasets targeted for the NASA Life Sciences Data Archive must pass a variety of quality checks to ensure data integrity and adherence to the standards defined by the LSDA Data Submission Guidelines (DSG) (see https://nlsp.nasa.gov/explore/lsdahome/datasubmit). These include requirements that data are consistent with open standards established by the omics community. Non-compliant data will not be accepted however archivists are available to advise submitters on how to revise data submissions and re-submit until compliance is achieved. Following compliance with the LSDA DSG, omics data next undergo a variety of additional quality checks to ensure the data meet omics community standards. Domain specific Omics data quality control tools and techniques are continually evolving and linked to the advancements in omics assays utilized and thus, the tools and techniques utilized by the LSDA for data quality control and validation will need to be sustained accordingly. All human omics data will be access controlled according to the policies described above, and requiring IRB approval for any additional access grants once the data are acquired (including access for analysis using the NLSP workspace tools).

Omics↗

DOE OSTI2016

SU-F-T-524: Investigation of the Dosimertric Benefits of Interchangeable Source Size of a Novel Rotating Gamma System

Purpose: Tremendous technological developments were made for conformal therapy techniques with linear accelerators, while less attention was paid to cobalt-60 units. The aim of the current study is to explore the dosimetric benefits of a novel rotating gamma ray system enhanced with interchangeable source sizes and multi-leaf collimator (MLC). Material and Methods: CybeRT is a novel rotating gamma ray machine with a ring gantry that ensures an iso-center accuracy of less than 0.3 mm. The new machine has a 70cm source axial distance allowing for improved penumbra compared to conventional machines. MCBEAM was used to simulate Cobalt-60 beams from the CybeRT head, while the MCPLAN code was used for modeling the MLC and for phantom/patient dose calculation. The CybeRT collimation will incorporate a system allowing for interchanging source sizes. In this work we have created phase space files for 1cm and 2cm source sizes. Evaluation of the system was done by comparing CybeRT beams with the 6MV beams in a water phantom and in patient geometry. Treatment plans were compared based on isodose distributions and dose volume histograms. Results: Profiles for the 1cm source were comparable to that from 6MV in the order of 6mm for 10×10 cm{sup 2} field size at the depth of maximum dose. This could ascribe to Cobalt-60 beams producing lowerenergy secondary electrons. Although, the 2cm source have a larger penumbra however it could be still used for large targets with proportionally increased dose rate. For large lung targets, the difference between cobalt and 6MV plans is clinically insignificant. Our preliminary results showed that interchanging source sizes will allow cobalt beams for volumetric arc therapy of both small lesions and large tumors. Conclusion: The CybeRT system will be a cost effective machine capable of performing advanced radiation therapy treatments of both small tumors and large target volumes.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2021

Reactive Transport Modeling of Aquifer Thermal Energy Storage System at Stockton, NJ

This is the modeling data (input/output files of TOUGHREACT 4.10) used to simulate the reactive transport processes of the Aquifer Thermal Energy Storage (ATES) operations at Stockton University, NJ. Readme.txt lists all the files. TOUGHREACT 4.10 requires to reproduce the modeling output. The modeling data in this submission is related to the Aquifer Injection for Energy Storage purposes outlined in "Reactive Transport Modeling of Aquifer Thermal Energy Storage System at Stockton, NJ During Seasonal Operations".

15 GEOTHERMAL ENERGY↗

DOE OSTI2023

Pavilion 2 Feature Additions [Slides]

Pavilion 2 is a critical component of system testing for LANL HPC systems. Pavilion 2 is system independent, which allows for the creation of a large suite of generalized tests than can be applied to multiple systems. These additions to Pavilion 2 improve our ability to test systems and diagnose issues. With such a large suite, running the proper tests when diagnosing and fixing a system can be difficult, as well as locating old tests for reference. Test configurations may contain several permutations of a test, not all of which may apply to a given system. The ability to run sets of test permutations lets the tester focus on specific parts of the test without having to edit the test configurations or run extraneous tests. The addition of the filter argument improves the ability to find test results based on test attributes, including past tests. Pavilion 2's system independence is largely based on various layers of configuration files. Configuration files for the host, test, and modes exist to allow generalized testing. The operating system configuration layer can provide helpful OS defaults to a test. Survey, a collection and reporting tool, can help us gauge the performance of a system as well as diagnose issues. The addition of a Survey mode in Pavilion 2 makes running tests with Survey very straightforward. The Survey results are combined with Pavilion 2's test results to make it easy to access.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Pavilion 2 Feature Additions [Slides]

Pavilion 2 is a critical component of system testing for LANL HPC systems. Pavilion 2 is system independent, which allows for the creation of a large suite of generalized tests than can be applied to multiple systems. These additions to Pavilion 2 improve our ability to test systems and diagnose issues. With such a large suite, running the proper tests when diagnosing and fixing a system can be difficult, as well as locating old tests for reference. Test configurations may contain several permutations of a test, not all of which may apply to a given system. The ability to run sets of test permutations lets the tester focus on specific parts of the test without having to edit the test configurations or run extraneous tests. The addition of the filter argument improves the ability to find test results based on test attributes, including past tests. Pavilion 2's system independence is largely based on various layers of configuration files. Configuration files for the host, test, and modes exist to allow generalized testing. The operating system configuration layer can provide helpful OS defaults to a test. Survey, a collection and reporting tool, can help us gauge the performance of a system as well as diagnose issues. The addition of a Survey mode in Pavilion 2 makes running tests with Survey very straightforward. The Survey results are combined with Pavilion 2's test results to make it easy to access.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS2000

BOREAS RSS-10 TOMS Circumpolar One-Degree PAR Images

The Boreal Ecosystem-Atmosphere Study (BOREAS) Remote Sensing Science (RSS)-10 team investigated the magnitude of daily, seasonal, and yearly variations of Photosynthetically Active Radiation (PAR) from ground and satellite observations. This data set contains satellite estimates of surface-incident PAR (400-700 nm, MJ/sq m) at one-degree spatial resolution. The spatial coverage is circumpolar from latitudes of 41 to 66 degrees north. The temporal coverage is from May through September for years 1979 through 1989. Eleven-year statistics are also provided: (1) mean, (2) standard deviation, and (3) coefficient of variation for 1979-89. The PAR estimates were derived from the global gridded ultraviolet reflectivity data product (average of 360, 380 nm) from the Nimbus-7 Total Ozone Mapping Spectrometer (TOMS). Image mask data are provided for identifying the boreal forest zone, and ocean/land and snow/ice-covered areas. The data are available as binary image format data files. The PAR data are available from the Earth Observing System Data and Information System (EOSDIS) Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC). The data files are available on a CD-ROM (see document number 20010000884).

Dye, Dennis G.↗

NASA NTRS2000

BOREAS HYD-2 Estimated Snow Water Equivalent (SWE) from Microwave Measurements

The surface meteorological data collected at the Boreal Ecosystem-Atmosphere Study (BOREAS) tower and ancillary sites are being used as inputs to an energy balance model to monitor the amount of snow storage in the boreal forest region. The BOREAS Hydrology (HYD)-2 team used Snow Water Equivalent (SWE) derived from an energy balance model and in situ observed SWE to compare the SWE inferred from airborne and spaceborne microwave data, and to assess the accuracy of microwave retrieval algorithms. The major external measurements that are needed are snowpack temperature profiles, in situ snow areal extent, and SWE data. The data in this data set were collected during February 1994 and cover portions of the Southern Study Area (SSA), Northern Study Area (NSA), and the transect areas. The data are available from BORIS as comma-delimited tabular ASCII files. The SWE data are available from the Earth Observing System Data and Information System (EOSDIS) Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC). The data files are available on a CD-ROM (see document number 20010000884).

Powell, Hugh↗

NASA NTRS2004

Telescience Resource Kit Software Capabilities and Future Enhancements

The Telescience Resource Kit (TReK) is a suite of PC-based software applications that can be used to monitor and control a payload on board the International Space Station (ISS). This software provides a way for payload users to operate their payloads from their home sites. It can be used by an individual or a team of people. TReK provides both local ground support system services and an interface to utilize remote services provided by the Payload Operations Integration Center (POIC). by the POIC and to perform local data functions such as processing the data, storing it in local files, and forwarding it to other computer systems. TReK can also be used to build, send, and track payload commands. In addition to these features, work is in progress to add a new command management capability. This capability will provide a way to manage a multi- platform command environment that can include geographically distributed computers. This is intended to help those teams that need to manage a shared on-board resource such as a facility class payload. The environment can be configured such that one individual can manage all the command activities associated with that payload. This paper will provide a summary of existing TReK capabilities and a description of the new command management capability. For example, 7'ReK can be used to receive payload data distributed

Schneider, Michelle↗

DOE OSTI2014

Twitter Stream Archiver

The Twitter Archiver system allows a user to enter their Twitter developer account credentials (obtained separately from the Twitter developer website) and read from the freely available Twitter sample stream. The Twitter sample stream provides a random sample of the overall volume of tweets that are contributed by users to the system. The Twitter Archiver system consumes the stream and serializes the information to text files at some predefined interval. A separate utility reads the text files and creates a searchable index using the open source Apache Lucene text indexing system.

Steed, Chad Allen↗