Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “small files”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS1999

Internet Protocol Enhanced over Satellite Networks

Extensive research conducted by the Satellite Networks and Architectures Branch of the NASA Lewis Research Center led to an experimental change to the Internet's Transmission Control Protocol (TCP) that will increase performance over satellite channels. The change raises the size of the initial burst of data TCP can send from 1 packet to 4 packets or roughly 4 kilobytes (kB), whichever is less. TCP is used daily by everyone on the Internet for e-mail and World Wide Web access, as well as other services. TCP is one of the feature protocols used in computer communications for reliable data delivery and file transfer. Increasing TCP's initial data burst from the previously specified single segment to approximately 4 kB may improve data transfer rates by up to 27 percent for very small files. This is significant because most file transfers in wide-area networks today are small files, 4 kilobytes or less. In addition, because data transfers over geostationary satellites can take 5 to 20 times longer than over typical terrestrial connections, increasing the initial burst of data that can be sent is extremely important. This research along with research from other institutions has led to the release of two new Request for Comments from the Internet Engineering Task Force (IETF, the international body that sets Internet standards). In addition, two studies of the implications of this mechanism were also funded by NASA Lewis.

Ivancic, William D.↗
NASA NTRS1992

Zebra: A striped network file system

The design of Zebra, a striped network file system, is presented. Zebra applies ideas from log-structured file system (LFS) and RAID research to network file systems, resulting in a network file system that has scalable performance, uses its servers efficiently even when its applications are using small files, and provides high availability. Zebra stripes file data across multiple servers, so that the file transfer rate is not limited by the performance of a single server. High availability is achieved by maintaining parity information for the file system. If a server fails its contents can be reconstructed using the contents of the remaining servers and the parity information. Zebra differs from existing striped file systems in the way it stripes file data: Zebra does not stripe on a per-file basis; instead it stripes the stream of bytes written by each client. Clients write to the servers in units called stripe fragments, which are analogous to segments in an LFS. Stripe fragments contain file blocks that were written recently, without regard to which file they belong. This method of striping has numerous advantages over per-file striping, including increased server efficiency, efficient parity computation, and elimination of parity update.

Hartman, John H.↗
NASA NTRS1993

Using Transparent Informed Prefetching (TIP) to reduce file read latency

As processor performance gains continue to outstrip Input/Output gains, I/O performance is becoming critical to overall system performance. File read latency is the most significant bottleneck for high performance I/O. Other aspects of I/O performance benefit from recent advances in disk bandwidth and throughput resulting from disk arrays, and in write performance derived from buffered write behind and the Log-structured File System. The access gap problem limiting improvements in read latency is exacerbated by distributed file systems operating over networks with diverse bandwidth. Focus is on extending the power of caching and prefetching to reduce file read latencies by exploiting hints from high-levels of a system. Such Transparent Informed Prefetching, TIP, and its benefits are described. It is argued that hints that disclose high level knowledge are a means for transferring optimization information across, without violating, module boundaries. How TIP can be used to convert the high throughput of new technologies such as disk arrays and log-structured file systems into low latency for applications is discussed. Our preliminary experiments show reductions in wall - clock execution time of 13 percent and 20 percent for a multiple module compilation tool (make) accessing data on a local disk and remote Coda file server, respectively, and a reduction of 30 percent for a text search (grep) remotely accessing many small files.

Patterson, R. H.↗
NASA NTRSDate not supplied

Acquisition of and Access to Research Omics Data

Omics data are essential for understanding the myriad and complex effects of space environments on humans. To assure maximum benefit from these kinds of data, the NASA Human Research Program Data Management Plan stipulates that human omics data should be archived within and accessed through the NASA Life Sciences Portal (NLSP). The NLSP has the capability to acquire and provision access to omics (and other kinds of) research results for individual and ad-hoc groups of subjects at the direction of institutional review boards, or other authorizing bodies or individuals, per institutional, program and investigation-specific policies and procedures. However, because some single-subject omics data, like CT scans and other kinds of large, complex biomedical data, could be used to identify heretofore unknown risks to the subject’s health, or, in certain cases, be used to identify a subject, NASA Policy Directive 7170.1 describes various policies regarding the management of and access to “research genetic testing” data, which includes many kinds of omics data. For example, NPD 7170.1 prohibits access to human research genetic data by NASA personnel who make employment decisions for the subjects from whom the data were obtained. To meet the objective of acquiring research omics data for NLSP in compliance with the policies in NPD 7170.1 and other applicable NASA policies, we designed NOMADS (the NLSP Omics Multimodal Acquisition of Data System), a new component that supports the transfer of large research data files, including research genetic testing data, using one of several different transfer mechanisms. The choice of mechanism is made by the submitter of the data, with guiding information from the system, and is likely to often be determined in large part by the nature and source location of the data. For example, for small files where the source data files are not already stored in a cloud storage system, users are likely to prefer to transfer their data to the NLSP via a web browser. Conversely, for large sets of files already organized and stored in a cloud storage system, users may opt for NOMAD’s cloud-to-cloud transfer method. All omics datasets targeted for the NASA Life Sciences Data Archive must pass a variety of quality checks to ensure data integrity and adherence to the standards defined by the LSDA Data Submission Guidelines (DSG) (see https://nlsp.nasa.gov/explore/lsdahome/datasubmit). These include requirements that data are consistent with open standards established by the omics community. Non-compliant data will not be accepted however archivists are available to advise submitters on how to revise data submissions and re-submit until compliance is achieved. Following compliance with the LSDA DSG, omics data next undergo a variety of additional quality checks to ensure the data meet omics community standards. Domain specific Omics data quality control tools and techniques are continually evolving and linked to the advancements in omics assays utilized and thus, the tools and techniques utilized by the LSDA for data quality control and validation will need to be sustained accordingly. All human omics data will be access controlled according to the policies described above, and requiring IRB approval for any additional access grants once the data are acquired (including access for analysis using the NLSP workspace tools).

Omics↗
NASA NTRS2024

Acquisition of and Access to Research Omics Data

Omics data are essential for understanding the myriad and complex effects of space environments on humans. To assure maximum benefit from these kinds of data, the NASA Human Research Program Data Management Plan stipulates that human omics data should be archived within and accessed through the NASA Life Sciences Portal (NLSP). The NLSP has the capability to acquire and provision access to omics (and other kinds of) research results for individual and ad-hoc groups of subjects at the direction of institutional review boards, or other authorizing bodies or individuals, per institutional, program and investigation-specific policies and procedures. However, because some single-subject omics data, like CT scans and other kinds of large, complex biomedical data, could be used to identify heretofore unknown risks to the subject’s health, or, in certain cases, be used to identify a subject, NASA Policy Directive 7170.1 describes various policies regarding the management of and access to “research genetic testing” data, which includes many kinds of omics data. For example, NPD 7170.1 prohibits access to human research genetic data by NASA personnel who make employment decisions for the subjects from whom the data were obtained. To meet the objective of acquiring research omics data for NLSP in compliance with the policies in NPD 7170.1 and other applicable NASA policies, we designed NOMADS (the NLSP Omics Multimodal Acquisition of Data System), a new component that supports the transfer of large research data files, including research genetic testing data, using one of several different transfer mechanisms. The choice of mechanism is made by the submitter of the data, with guiding information from the system, and is likely to often be determined in large part by the nature and source location of the data. For example, for small files where the source data files are not already stored in a cloud storage system, users are likely to prefer to transfer their data to the NLSP via a web browser. Conversely, for large sets of files already organized and stored in a cloud storage system, users may opt for NOMAD’s cloud-to-cloud transfer method. All omics datasets targeted for the NASA Life Sciences Data Archive must pass a variety of quality checks to ensure data integrity and adherence to the standards defined by the LSDA Data Submission Guidelines (DSG) (see https://nlsp.nasa.gov/explore/lsdahome/datasubmit). These include requirements that data are consistent with open standards established by the omics community. Non-compliant data will not be accepted however archivists are available to advise submitters on how to revise data submissions and re-submit until compliance is achieved. Following compliance with the LSDA DSG, omics data next undergo a variety of additional quality checks to ensure the data meet omics community standards. Domain specific Omics data quality control tools and techniques are continually evolving and linked to the advancements in omics assays utilized and thus, the tools and techniques utilized by the LSDA for data quality control and validation will need to be sustained accordingly. All human omics data will be access controlled according to the policies described above, and requiring IRB approval for any additional access grants once the data are acquired (including access for analysis using the NLSP workspace tools).

Omics↗
NASA NTRS2016

Human Flight to Lunar and Beyond - Re-Learning Operations Paradigms

For the first time since the Apollo era, NASA is planning on sending astronauts on flights beyond LEO. The Human Space Flight (HSF) program started with a successful initial flight in Earth orbit, in December 2014. The program will continue with two Exploration Missions (EM): EM-1 will be unmanned and EM-2, carrying astronauts, will follow. NASA established a multi-center team to address the communications, and related tacking/navigation needs. This paper will focus on the lessons learned by the team designing the architecture and operations for the missions. Many of these Beyond Earth Orbit lessons had to be re-learned, as the HSF program has operated for many years in Earth orbit. Unlike the Apollo missions that were largely tracked by a dedicated ground network, the HSF planned missions will be tracked (at distances beyond GEO) by the DSN, a network that mostly serves robotic missions. There have been surprising challenges to the DSN as unique modern human spaceflight needs stretch the experience base beyond that of tracking robotic missions in deep space. Close interaction between the DSN and the HSF community to understand the unique needs (e.g. 2-way voice) resulted in a Concept of Operations (ConOps) that leverages both the deep space robotic and the Human LEO experiences. Several examples will be used to highlight the unique challenges the team faced in establishing the communications and tracking capabilities for HSF missions beyond Earth Orbit, including: Navigation. At LEO, HSF missions can rely on GPS devices for orbit determination. For Lunar-and-beyond HSF missions, techniques such as precision 2-way and 3-way Doppler and ranging, Delta-Difference-of-range, and eventually possibly on-board navigation will be used. At the same time, HSF presents a challenge to navigators, beyond those presented by robotic missions - navigating a dynamic/"noisy" spacecraft. Impact of latency - the delay associated with Round-Trip-Light-Time (RTLT). Imagine trying to have a 2-way discussion (audio or video) with an astronaut, with a 2-3 sec or more delay inserted (for lunar distances) or 20 minutes delay (for Mars distances). Balanced communications link. For robotic missions, there has been a heavy emphasis on higher downlink data rates, e.g. bringing back science data. Higher uplink data rates were of secondary importance, as uplink was used only to send commands (and occasionally small files) to the spacecraft. The ratio of downlink-to-uplink data rates was often 10:1 or more. For HSF, a continuous forward link is established and rates for uplink and downlink are more similar.

Kenny, Edward (Ted)↗
NASA NTRS1987

Usage analysis of user files in UNIX

Presented is a user-oriented analysis of short term file usage in a 4.2 BSD UNIX environment. The key aspect of this analysis is a characterization of users and files, which is a departure from the traditional approach of analyzing file references. Two characterization measures are employed: accesses-per-byte (combining fraction of a file referenced and number of references) and file size. This new approach is shown to distinguish differences in files as well as users, which cam be used in efficient file system design, and in creating realistic test workloads for simulations. A multi-stage gamma distribution is shown to closely model the file usage measures. Even though overall file sharing is small, some files belonging to a bulletin board system are accessed by many users, simultaneously and otherwise. Over 50% of users referenced files owned by other users, and over 80% of all files were involved in such references. Based on the differences in files and users, suggestions to improve the system performance were also made.

Devarakonda, Murthy V.↗
NASA NTRS1993

Software to model AXAF image quality

This draft final report describes the work performed under this delivery order from May 1992 through June 1993. The purpose of this contract was to enhance and develop an integrated optical performance modeling software for complex x-ray optical systems such as AXAF. The GRAZTRACE program developed by the MSFC Optical Systems Branch for modeling VETA-I was used as the starting baseline program. The original program was a large single file program and, therefore, could not be modified very efficiently. The original source code has been reorganized, and a 'Make Utility' has been written to update the original program. The new version of the source code consists of 36 small source files to make it easier for the code developer to manage and modify the program. A user library has also been built and a 'Makelib' utility has been furnished to update the library. With the user library, the users can easily access the GRAZTRACE source files and build a custom library. A user manual for the new version of GRAZTRACE has been compiled. The plotting capability for the 3-D point spread functions and contour plots has been provided in the GRAZTRACE using the graphics package DISPLAY. The Graphics emulator over the network has been set up for programming the graphics routine. The point spread function and the contour plot routines have also been modified to display the plot centroid, and to allow the user to specify the plot range, and the viewing angle options. A Command Mode version of GRAZTRACE has also been developed. More than 60 commands have been implemented in a Code-V like format. The functions covered in this version include data manipulation, performance evaluation, and inquiry and setting of internal parameters. The user manual for these commands has been formatted as in Code-V, showing the command syntax, synopsis, and options. An interactive on-line help system for the command mode has also been accomplished to allow the user to find valid commands, command syntax, and command function. A translation program has been written to convert FEA output from structural analysis to GRAZTRACE surface deformation file (.dfm file). The program can accept standard output files and list files from COSMOS/M and NASTRAN finite analysis programs. Some interactive options are also provided, such as Cartesian or cylindrical coordinate transformation, coordinate shift and scale, and axial length change. A computerized database for technical documents relating to the AXAF project has been established. Over 5000 technical documents have been entered into the master database. A user can now rapidly retrieve the desired documents relating to the AXAF project. The summary of the work performed under this contract is shown.

Ahmad, Anees↗
NASA NTRS1990

Solar Astronomy Data Base: Packaged Information on Diskette

In its role as a library, the National Geophysical Data Center has transferred to diskette a collection of small, digital files of routinely measured solar indices for use on an IBM-compatible desktop computer. Recording these observations on diskette allows the distribution of specialized information to researchers with a wide range of expertise in computer science and solar astronomy. Every data set was made self-contained by including formats, extraction utilities, and plain-language descriptive text. Moreover, for several archives, two versions of the observations are provided - one suitable for display, the other for analysis with popular software packages. Since the files contain no control characters, each one can be modified with any text editor.

Mckinnon, John A.↗
NASA NTRS1991

An area model for on-chip memories and its application

An area model suitable for comparing data buffers of different organizations and arbitrary sizes is described. The area model considers the supplied bandwidth of a memory cell and includes such buffer overhead as control logic, driver logic, and tag storage. The model gave less than 10 percent error when verified against real caches and register files. It is shown that, comparing caches and register files in terms of area for the same storage capacity, caches generally occupy more area per bit than register files for small caches because the overhead dominates the cache area at these sizes. For larger caches, the smaller storage cells in the cache provide a smaller total cache area per bit than the register set. Studying cache performance (traffic ratio) as a function of area, it is shown that, for small caches, direct-mapped caches perform significantly better than four-way set-associative caches and, for caches of medium areas, both direct-mapped and set-associative caches perform better than fully associative caches.

Mulder, Johannes M.↗
NASA NTRS2011

Commercialization of LARC (TradeMark) -SI Polyimide Technology

LARC(TradeMark)-SI, Langley Research Center- Soluble Imide, was developed in 1992, with the first patent issuing in 1997, and then subsequent patents issued in 1998 and 2000. Currently, this polymer has been successfully licensed by NASA, and has generated revenues, at the time of this reporting, in excess of $1.4 million. The success of this particular polymer has been due to many factors and many lessons learned to the point that the invention, while important, is the least significant part in the commercialization of this material. Commercial LARC(TradeMark)-SI is a polyimide composed of two molar equivalents of dianhydrides: 4,4 -oxydiphthalic anhydride (ODPA), and 3,3 ,4,4 -biphenyltetracarboxylic dianhydride (BPDA) and 3,4 -oxydianiline (3,4 -ODA) as the diamine. The unique feature of this aromatic polyimide is that it remains soluble after solution imidization in high-boiling, polar aprotic solvents, even at solids contents of 50-percent by weight. However, once isolated and heated above its T(sub g) of 240 C, it becomes insoluble and exhibits high-temperature thermoplastic melt-flow behavior. With these unique structure property characteristics, it was thought this would be an advantage to have an aromatic polyimide that is both solution and melt processable in the imide form. This could potentially lead to lower cost production as it was not as equipment- or labor-intensive as other high-performance polyimide materials that either precipitate or are intractable. This unique combination of properties allowed patents with broad claim coverage and potential commercialization. After the U.S. Patent applications were filed, a Small Business Innovation Research (SBIR) contract was awarded to Imtec, Inc. to develop and supply the polyimide to NASA and the general public. Some examples of demonstration parts made with LARC(TradeMark)-SI ranged from aircraft wire and multilayer printed-circuit boards, to gears, composite panels, supported adhesive tape, composite coatings, cookware, and polyimide foam. Even with its unique processing characteristics, the thermal and mechanical properties were not drastically different from other solution or meltprocessable polyimides developed by NASA. LARC(TradeMark)-SI risked becoming another interesting, but costly, high-performance material.

Bryant, Robert G.↗
NASA NTRS2019

Providing Data Access and Analysis Capabilities to SERVIR’s Data-Sparse Regions

In developing regions of the world, the communications infrastructure pose enormous challenges for using Earth observation data. Limited internet bandwidth along with the high costs make it almost impossible to process and extract zonal statistics over large periods of time for even small geographic areas. In such cases, downloading daily rainfall data or dekadal series of NDVI data would take days and consume all the bandwidth allocated to an organization (for reference, internet connections in Niger would cost thousands of dollars per month at a maximum - and unreliable - bandwidth of just 10 Mbps). Running crop models or hydrological models typically require several years of historic data over the area of interest (AOI). In some cases, these AOIs are relatively small compared to the footprint of individual earth observation granules. Hence, systems that let the stakeholders subset the data to download to a user specified area, or even submit processing requests that let them download small result files for the AOI become critical. The SERVIR program has developed a tool to provide this type of access to help decision makers in developing regions use long time series of adjusted rainfall data (CHIRPS), NDVI values, seasonal weather forecasts, evaporative stress indices and others in a very efficient manner. This system, named ClimateSERV (https://climateserv.servirglobal.net) ingests the datasets in an automated fashion and allows interactive access (through a web application), or automated access through a simple API that developers can quickly incorporate in independent applications. This way, the extraction of daily averages of rainfall over a 50 square Km area through 30 years of archived data takes only a few seconds to process, and the results can be presented on an online chart or downloaded in a comma separated file that's only a few Kb.

Ashmall, William↗
NASA NTRSDate not supplied

The Anatomy of Software Changes and Bugs in Autonomous Operating System

Cyberphysical systems with autonomous functions are complex pieces of software, consisting of many components, some of which implement autonomous functionality and some may use AI or machine learning algorithms. Software bugs in an autonomous system are of particular concern, as they can have catastrophic consequences. However, detailed studies based on empirical data are rare and therefore these bugs are not well understood. This paper aims to contribute towards filling that gap by investigating the software changes and bugs in Autonomy Operating System (AOS) for Unmanned Aircraft Systems (UAS), which consist of 26 components containing about 103,000 lines of code and having a total of 772 bugfixes. Based on the data extracted from the code repository and semi-structured interviews with the developers of AOS, we explore the differences among autonomous software components, components developed using Model-based Software Engineering, and reuse with respect to change proneness, fault proneness, distribution of bugfixes among AOS components and files of these components, and characteristics of bugs of different AOS components. Our results show that the autonomous components were significantly more change prone (measured in number of commits and code churn) and fault prone (measured in bugfixes per KLoC) than non-autonomous components. The distribution of the locations of bugfixes was skewed, both at component and file level (i.e., a small number of components / files contained the majority of bugs). These evidence-based findings provide important insights to researchers and practitioners alike and can be used to efficiently improve the quality and reliability of autonomous systems.

Katerina Goseva-Popstojanova↗
NASA NTRS2023

Application of Fountain Code to High-Rate Delay Tolerant Networks

Space communication poses several unique challenges that are not always present in typical terrestrial communications. Currently, communication with satellites is based on point-to-point links, and development of an interplanetary internet is an active research area. Delay Tolerant Networking (DTN) has been proposed as a way to mitigate the long delays and disruptions found in deep space. A specialized version of DTN, called High-rate Delay Tolerant Networking (HDTN), has been developed by NASA to support a variety of missions requiring store-and-forward capability. However, there are still several features that are desired for HDTN including data fragmentation, multicast, and anycast. This project proposes the application of fountain code in HDTN as a means of fragmenting, distributing, and reassembling data (in the form of bundles) across multiple nodes (i.e. satellites) to any number of receivers (i.e. ground stations). Fountain code is shown to be a promising encoding method for use with the HDTN protocol suite due to its short runtimes, small encoded file sizes, and loss tolerance.

Noah Douglass↗
NASA NTRS2022

Automatic variable selection in ecological niche modeling: A case study using Cassin’s Sparrow (Peucaea cassinii)

MERRA/Max provides a feature selection approach to dimensionality reduction that enables direct use of global climate model outputs in ecological niche modeling. The system accomplishes this reduction through a Monte Carlo optimization in which many independent MaxEnt runs, operating on a species occurrence file and a small set of randomly selected variables in a large collection of variables, converge on an estimate of the top contributing predictors in the larger collection. These top predictors can be viewed as potential candidates in the variable selection step of the ecological niche modeling process. MERRA/Max’s Monte Carlo algorithm operates on files stored in the underlying filesystem, making it scalable to large data sets. Its software components can run as parallel processes in a high-performance cloud computing environment to yield near real-time performance. In tests using Cassin’s Sparrow (Peucaea cassinii) as the target species, MERRA/Max selected a set of predictors from Worldclim’s Bioclim collection of 19 environmental variables that have been shown to be important determinants of the species’ bioclimatic niche. It also selected biologically and ecologically plausible predictors from a more diverse set of 86 environmental variables derived from NASA’s Modern-Era Retrospective Analysis for Research and Applications Version 2 (MERRA-2) reanalysis, an output product of the Goddard Earth Observing System Version 5 (GEOS-5) modeling system. We believe these results point to a technological approach that could expand the use global climate model outputs in ecological niche modeling, foster exploratory experimentation with otherwise difficult-to-use climate data sets, streamline the modeling process, and, eventually, enable automated bioclimatic modeling as a practical, readily accessible, low-cost, commercial cloud service.

John L. Schnase↗
NASA NTRS2002

Creating a Team Archive During Fast-Paced Anomaly Response Activities in Space Missions

This paper describes a Web-based system to support the temporary Anomaly Response Team formed from distributed subteams in Space Shuttle and International Space Station missions. The system was designed for easy and flexible creation of small collections of files and links associated with work on a particular anomaly. The system supports privacy and levels of formality for the subteams. First we describe the supported groups and an anomaly response scenario. Then we describe the support system prototype, the Anomaly Response Tracking and Integration System (ARTIS). Finally, we describe our evaluation approach and the results of the evaluation.

Malin, Jane T.↗
NASA NTRSDate not supplied

Multidimensional Data Aggregation in the Cloud with Application to Geostationary Satellite-based Air Quality Monitoring

Scientists use satellite data for studying Earth's systems, and the remote sensing data that these satellites collect are typically separated into files of a size small enough for efficient network transfer and storage. However, researchers usually prefer to analyze the data based on real-world dimensions like time, space, or elevation. To help with this, NASA's Atmospheric Science Data Center (ASDC) developed a new cloud-based tool that combines these smaller data chunks into larger, more useful datasets. The tool works on Network Common Data Form (netCDF4) and some HDF5 formatted files, and it is available as a service in NASA's Earthdata Cloud. In this presentation, we showcase this service using data from the Tropospheric Emissions: Monitoring of Pollution (TEMPO) instrument. By combining TEMPO's continuous observations over time, we create longer and more informative analysis-ready time series to facilitate the study of air quality patterns. Insights gained will provide a more comprehensive understanding of pollution sources, transport patterns, and their effects on the environment and human health.

Daniel Kaufman↗
NASA NTRS2009

Low-Complexity Lossless and Near-Lossless Data Compression Technique for Multispectral Imagery

This work extends the lossless data compression technique described in Fast Lossless Compression of Multispectral- Image Data, (NPO-42517) NASA Tech Briefs, Vol. 30, No. 8 (August 2006), page 26. The original technique was extended to include a near-lossless compression option, allowing substantially smaller compressed file sizes when a small amount of distortion can be tolerated. Near-lossless compression is obtained by including a quantization step prior to encoding of prediction residuals. The original technique uses lossless predictive compression and is designed for use on multispectral imagery. A lossless predictive data compression algorithm compresses a digitized signal one sample at a time as follows: First, a sample value is predicted from previously encoded samples. The difference between the actual sample value and the prediction is called the prediction residual. The prediction residual is encoded into the compressed file. The decompressor can form the same predicted sample and can decode the prediction residual from the compressed file, and so can reconstruct the original sample. A lossless predictive compression algorithm can generally be converted to a near-lossless compression algorithm by quantizing the prediction residuals prior to encoding them. In this case, since the reconstructed sample values will not be identical to the original sample values, the encoder must determine the values that will be reconstructed and use these values for predicting later sample values. The technique described here uses this method, starting with the original technique, to allow near-lossless compression. The extension to allow near-lossless compression adds the ability to achieve much more compression when small amounts of distortion are tolerable, while retaining the low complexity and good overall compression effectiveness of the original algorithm.

Xie, Hua↗