SPLINE BASED TRANSFORMER

Spline-based transformer ML architecture (Bächer, robotics/motion adjacent).

In one embodiment, a method for generating an output sequence of data utilizing a spline-based transformer is disclosed. The method may include encoding, via a processing element, an input sequence of data using an artificial neural network encoder to generate a plurality of input tokens; processing, via the processing element, the plurality of input tokens and a plurality of control tokens with a transformer encoder into a latent space to generate a plurality of control points; defining, via the processing element, a spline based on the plurality of control points; sampling, via the processing element, a plurality of interpolated control points based on the spline; and decoding, via the processing element, the interpolated control points with an artificial neural network decoder to generate the output sequence of data.

BACKGROUND

Positional encoding is traditionally an important component in transformer models. It infuses positional information into input tokens to help transformers learn position-agnostic token embeddings. Positional encoding works by (1) pre-assigning sinusoids of different frequencies and phases to every position an input token can take on in a sequence, and (2) by adding this sinusoid to the token embedding that appears at the corresponding position in the sequence. Injecting a token with positional information, also referred to as absolute position encoding in later work, has evolved into several variants. For example, several works have shown that absolute position encoding limits the ability of transformers to handle longer sequences at inference time and proposed relative position encoding schemes where a fixed or learned bias is added to the attention matrix. Irrespective of their exact arrangement, transformer architectures generally employ a combination of absolute and relative position encoding schemes.

Positional encoding assumes that token embeddings represent elemental data in a collection, e.g., individual words in a sentence, images in a video, or poses in an animation, and that an additional notion of position is required to model a collection of such elements, such as sequences of words, images, or animation frames. This process decouples elemental and collective datatypes and forces a model to learn separate representations for the elements