Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “tokenization”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

Guaranteeing synchronous message deadlines with the timed token medium access control protocol

We study the problem of guaranteeing synchronous message deadlines in token ring networks where the timed token medium access control protocol is employed. Synchronous capacity, defined as the maximum time for which a node can transmit its synchronous messages every time it receives the token, is a key parameter in the control of synchronous message transmission. To ensure the transmission of synchronous messages before their deadlines, synchronous capacities must be properly allocated to individual nodes. We address the issue of appropriate allocation of the synchronous capacities. Several synchronous capacity allocation schemes are analyzed in terms of their ability to satisfy deadline constraints of synchronous messages. We show that an inappropriate allocation of the synchronous capacities could cause message deadlines to be missed even if the synchronous traffic is extremely low. We propose a scheme called the normalized proportional allocation scheme which can guarantee the synchronous message deadlines for synchronous traffic of up to 33 percent of available utilization. To date, no other synchronous capacity allocation scheme has been reported to achieve such substantial performance. Another major contribution of this paper is an extension to the previous work on the bounded token rotation time. We prove that the time elapsed between any consecutive visits to a particular node is bounded by upsilon TTRT, where TTRT is the target token rotation time set up at system initialization time. The previous result by Johnson and Sevcik is a special case where upsilon = 2. We use this result in the analysis of various synchronous allocation schemes. It can also be applied in other similar studies.

Agrawal, Gopal↗

WLCG Authorisation from X.509 to Tokens

The WLCG Authorisation Working Group was formed in July 2017 with the objective to understand and meet the needs of a future-looking Authentication and Authorisation Infrastructure (AAI) for WLCG experiments. Much has changed since the early 2000s when X.509 certificates presented the most suitable choice for authorisation within the grid; progress in token based authorisation and identity federation has provided an interesting alternative with notable advantages in usability and compatibility with external (commercial) partners. The need for interoperability in this new model is paramount as infrastructures and research communities become increasingly interdependent. Over the past two years, the working group has made significant steps towards identifying a system to meet the technical needs highlighted by the community during staged requirements gathering activities. Enhancement work has been possible thanks to externally funded projects, allowing existing AAI solutions to be adapted to our needs. A cornerstone of the infrastructure is the reliance on a common token schema in line with evolving standards and best practices, allowing for maximum compatibility and easy cooperation with peer infrastructures and services. We present the work of the group and an analysis of the anticipated changes in authorisation model by moving from X.509 to token based authorisation. A concrete example of token integration in Rucio is presented.

Bockelman, Brian↗

Tokenized Data for FORGE Foundation Models

This dataset comprises a vast corpus of 257 billion tokens, accompanied by the corresponding vocabulary file employed in the pre-training of FORGE foundation models. The primary data source for this corpus is scientific documents derived from diverse origins, and they have been tokenized using the Hugging Face BPE tokenizer. Further details about this research can be found in the publication titled FORGE: Pre-Training Open Foundation Models for Science authored by Junqi Yin, Sajal Dash, Feiyi Wang, and Mallikarjun (Arjun) Shankar, presented at SC'23. The data tokenization pipeline and resulting artifacts use CORE data [Ref: Knoth, P., and Zdrahal, Z. (2012). CORE: three access levels to underpin open access. D-Lib Magazine, 18(11/12)]. For use of these data sets for any purpose, please follow the guidelines provided in https://core.ac.uk/terms .

Yin, Junqi↗

Addressing tokens dynamic generation, propagation, storage and renewal to secure the GlideinWMS pilot based jobs and system

GlideinWMS has been one of the first middleware in the WLCG community to transition from X.509 to support also tokens. The first step was to get from the prototype in 2019 to using tokens in production in 2022. This paper will present the challenges introduced by the wider adoption of tokens and the evolution plans for securing the pilot infrastructure of GlideinWMS and supporting the new requirements. In the last couple of years, the GlideinWMS team supported the migration to tokens of experiments and resources. Inadequate support in the current infrastructure, more stringent requirements, and the higher spatial and temporal granularity forced GlideinWMS to revisit once more how credentials are generated, used, and propagated. The new credential modules have been designed to be used in multiple systems (GWMS, HC) and use a model where credentials have type, purpose, and different flows. Credentials are dynamically generated in order to customize the duration and limit the scope to the targeted resource. This allows to enforce the least privilege principle. Finally, we also considered adding credential storage, renewal, and invalidation mechanisms within the GlideinWMS infrastructure to serve better the experiments’ needs.

Mambelli, Marco↗

Addressing tokens dynamic generation, propagation, storage and renewal to secure the GlideinWMS pilot based jobs and system

GlideinWMS has been one of the first middleware in the WLCG community to transition from X.509 to support also tokens. The first step was to get from the prototype in 2019 to using tokens in production in 2022. This paper will present the challenges introduced by the wider adoption of tokens and the evolution plans for securing the pilot infrastructure of GlideinWMS and supporting the new requirements. In the last couple of years, the GlideinWMS team supported the migration of experiments and resources to tokens. Inadequate support in the current infrastructure, more stringent requirements, and the higher spatial and temporal granularity forced GlideinWMS to revisit once more how credentials are generated, used, and propagated. The new credential modules have been designed to be used in multiple systems (GlideinWMS, HEPCloud) and use a model where credentials have type, purpose, and different flows. Credentials are dynamically generated in order to customize the duration and limit the scope to the targeted resource. This allows to enforce the least privilege principle. Finally, we also considered adding credential storage, renewal, and invalidation mechanisms within the GlideinWMS infrastructure to better serve the experiments' needs.

Coimbra, Bruno Moreira [Fermilab] (ORCID:000900022↗

Performance comparison of token ring protocols for hard-real-time communication

The ability to guarantee the deadlines of synchronous messages while maintaining a good aggregate throughput is an important consideration in the design of distributed real-time systems. In this paper, we study two token ring protocols, the priority driven protocol and the timed token protocol, for their suitability for hard real-time systems. Both these protocols use a token to control access to the transmission medium. In a priority driven protocol, messages are assigned priorities and the protocol ensures that messages are transmitted in the order of their priorities. Timed token protocols do not provide for priority arbitration but ensure that the maximum access delay for a station is bounded. For both protocols, we first derive the schedulability conditions under which the transmission deadlines of a given set of synchronous messages can be guaranteed. Subsequently, we use these schedulability conditions to quantitatively compare the average case behavior of the protocols. This comparison demonstrates that each of the protocols has its domain of superior performance and neither dominates the other for the entire range of operating conditions.

Kamat, Sanjay↗

Throughput analysis of the IEEE 802.4 token bus standard under heavy load

It has become clear in the last few years that there is a trend towards integrated digital services. Parallel to the development of public Integrated Services Digital Network (ISDN) is service integration in the local area (e.g., a campus, a building, an aircraft). The types of services to be integrated depend very much on the specific local environment. However, applications tend to generate data traffic belonging to one of two classes. According to IEEE 802.4 terminology, the first major class of traffic is termed synchronous, such as packetized voice and data generated from other applications with real-time constraints, and the second class is called asynchronous which includes most computer data traffic such as file transfer or facsimile. The IEEE 802.4 token bus protocol which was designed to support both synchronous and asynchronous traffic is examined. The protocol is basically a timer-controlled token bus access scheme. By a suitable choice of the design parameters, it can be shown that access delay is bounded for synchronous traffic. As well, the bandwidth allocated to asynchronous traffic can be controlled. A throughput analysis of the protocol under heavy load with constant channel occupation of synchronous traffic and constant token-passing times is presented.

Pang, Joseph↗

System and method for key generation in security tokens

Functional randomness in security tokens (FRIST) may achieve improved security in two-factor authentication hardware tokens by improving on the algorithms used to securely generate random data. A system and method in one embodiment according to the present invention may allow for security of a token based on storage cost and computational security. This approach may enable communication where security is no longer based solely on onetime pads (OTPs) generated from a single cryptographic function (e.g., SHA-256).

97 MATHEMATICS AND COMPUTING↗

Is tokenization needed for masked particle modeling?

In this work, we significantly enhance masked particle modeling (MPM), a self-supervised learning scheme for constructing highly expressive representations of unordered sets relevant to developing foundation models for high-energy physics. In MPM, a model is trained to recover the missing elements of a set, a learning objective that requires no labels and can be applied directly to experimental data. We achieve significant performance improvements over previous work on MPM by addressing inefficiencies in the implementation and incorporating a more powerful decoder. We compare several pre-training tasks and introduce new reconstruction methods that utilize conditional generative models without data tokenization or discretization. We show that these new methods outperform the tokenized learning objective from the original MPM on a new test bed for foundation models for jets, which includes using a wide variety of downstream tasks relevant to jet physics, such as classification, secondary vertex finding, and track identification.

conditional generative models↗

ACTT: Automotive CAN Tokenization and Translation

Modern vehicles contain scores of Electrical Control Units (ECUs) that broadcast messages over a Controller Area Network (CAN). Vehicle manufacturers rely on security through obscurity by concealing their unique mapping of CAN messages to vehicle functions which differs for each make, model, year, and even trim. This poses a major obstacle for after-market modifications notably performance tuning and in-vehicle network security measures. We present ACTT: Automotive CAN Tokenization and Translation, a novel, vehicle-agnostic, algorithm that leverages available diagnostic information to parse CAN data into meaningful messages, simultaneously cutting binary messages into tokens, and learning the translation to map these contiguous bits to the value of the vehicle function communicated.

Verma, Miki↗

Distributed computing system with dual independent communications paths between computers and employing split tokens

This is a distributed computing system providing flexible fault tolerance; ease of software design and concurrency specification; and dynamic balance of the loads. The system comprises a plurality of computers each having a first input/output interface and a second input/output interface for interfacing to communications networks each second input/output interface including a bypass for bypassing the associated computer. A global communications network interconnects the first input/output interfaces for providing each computer the ability to broadcast messages simultaneously to the remainder of the computers. A meshwork communications network interconnects the second input/output interfaces providing each computer with the ability to establish a communications link with another of the computers bypassing the remainder of computers. Each computer is controlled by a resident copy of a common operating system. Communications between respective ones of computers is by means of split tokens each having a moving first portion which is sent from computer to computer and a resident second portion which is disposed in the memory of at least one of computer and wherein the location of the second portion is part of the first portion. The split tokens represent both functions to be executed by the computers and data to be employed in the execution of the functions. The first input/output interfaces each include logic for detecting a collision between messages and for terminating the broadcasting of a message whereby collisions between messages are detected and avoided.

Rasmussen, Robert D.↗

Documentation for the token ring network simulation system

A manual is presented which describes the language features of the Token Ring Network Simulation System. The simulation system is a powerful simulation tool for token ring networks which allows the specification of various Medium Access Control (MAC) layer protocols as well as the specification of various features of upper layer ISO protocols. In addition to these features, it also allows the user to specify message and station classes virtually to any degree of detail desired. The choice of a language instead of an interactive system to specify network parameters was dictated by both flexibility and time considerations. The language was developed specifically for the simulation system, and is very simple. It is also user friendly in that language elements which do not apply to the case at hand are ignored rather than treated as errors.

Peden, Jeffery H.↗

Advance Reservation Access Control Using Software-Defined Networking and Tokens

Advance reservation systems allow users to reserve dedicated bandwidth connection resources from an advanced high-speed network. A common use case for such systems is data transfers in distributed science environments in which a user wants exclusive access to the reservation. However, current advance network reservation methods cannot ensure exclusive access of a network reservation to the specific flow for which the user made the reservation. We present here a novel network architecture that addresses this limitation and ensures that a reservation is used only by the intended flow. We achieve this by leveraging software-defined networking (SDN) and token-based authorization. We use SDN to orchestrate and automate the reservation of networking resources, end-to-end and across multiple administrativedomains, and tokens to create a strong binding between the user or application that requested the reservation and the flows provisioned by SDN.

Kettimuthu, R.↗

Cache utilization and eviction based on allocated priority tokens

A system, method, and computer program product are provided for providing prioritized access for multithreaded processing. The method includes the steps of allocating threads to process a workload and assigning a set of priority tokens to at least a portion of the threads. Access to a resource, by each one of the threads, is based on the priority token assigned to the thread and the threads are executed by a multithreaded processor to process the workload.

Johnson, Daniel Robert↗

Conflux – An Asynchronous 2-to-1 Multiplexor for Time-Division Multiplexing and Clock-less, Token-less Readout

This paper describes an independent, self-arbitrating asynchronous two-to-one multiplexor called Conflux. Conflux can be used to implement clock-less and token-less Time-Division Multiplexing between two sources. This capability can then be used to create a clock-less and token-less aggregate data bus which can be used as a complete asynchronous readout architecture for a chip, a part of a chip or an entire multi-chip system. Conflux utilizes a classic, four-phased asynchronous handshake on both of its input ports as well as on its output port. Asynchronous Request is bundled with Data to ensure consistent propagation delay. The Request is also differential to account for propagation delay differences between Logical Ones and Zeros. Arbitration between the two Conflux input streams is accomplished by three Set-Reset latches. Finally, Conflux cells were used successfully to implement the readout architecture of the FCP130 prototype chip. Tests indicate single Conflux stage delays of 1.8ns and a bandwidth of approximately 20 Gbps in 130nm CMOS.

43 PARTICLE ACCELERATORS↗

Secure Command Line Solution for Token-based Authentication

The WLCG is modernizing its security infrastructure, replacing X.509 client authentication with the newer industry standard of JSON Web Tokens (JWTs) obtained through the Open ID Connect (OIDC) protocol. There is a wide variety of software available using the standards, but most of it is for Web browser-based applications and doesn’t adapt well to the command line-based software used heavily in High Throughput Computing (HTC). OIDC command line client software did exist, but it did not meet our requirements for security and convenience. This paper discusses a command line solution we have made based on the popular existing secrets management software from Hashicorp called vault. We made a package called htvault-config to easily configure a vault service and another called htgettoken to be the vault client. In addition, we have integrated use of the tools into the HTCondor workload management system, although they also work well independent of HTCondor. All of the software is open source, under active development, and ready for use.

Dykstra, Dave↗

A Priority Protocol for Token-Ring Networks

New priority protocol controls access to token-ring local-area network (LAN) of digital-communication stations over widely ranging mix of low- and high-priority traffic. Protocol, called round-robin priority scheme (RRPS), introduces only small overhead and therefore degrades system performance only minimally. Key messages guaranteed access to local-area network during peak loads.

Liu, H. T.↗

Fairness of channel access for non-time-critical traffic using the FDDI token ring protocol

The Fiber Distributed Data Interface (FDDI) is an ANSI draft proposed standard for a 100 megabit per second fiber optic token ring. FDDI supports two types of traffic, synchronous and asynchronous. Synchronous traffic is time critical traffic; stations are assigned guaranteed bandwidth to support their synchronous needs. Asynchronous traffic is lower priority and is sent only if time permits. It is proved analytically that the FDDI access protocol provides all stations on the ring with equal access to the channel to transmit asynchronous frames, regardless of the relative sizes of synchronous bandwidth allocations for individual stations. Analytic results are supported with data from simulation runs.

Johnson, M. J.↗