Skip to main content

Tokenize

POST 
/tokenize/:reader_id

Tokenize

The tokenization endpoint takes in a text, and returns the tokenized version of the text: the way the text is seen by the model with the path-supplied reader_id. Provide either a single text, or a pair of texts to be tokenized together (as you would a chat template). Does not support batch inputs.

The reader_id can be found as the model's key in the live_readers object, returned by the status endpoint. It can also be supplied at boot time for each model in the config: see here for more information.

Request​

Path Parameters

    reader_id stringrequired

    The ID of the reader to use for tokenization.

Body

required

    skip_special_tokensbooleannullable

    text

    object

    required

    Input Text used for ease of users not to have to use the clunky PayloadText. Mapping provided below to convert InputText to PayloadText.

    oneOf

    string

    use_chat_templatebooleannullable

Responses​

Takes in a JSON payload and returns the response all at once.

Schema

    skip_special_tokensbooleannullable

    text

    object

    required

    Input Text used for ease of users not to have to use the clunky PayloadText. Mapping provided below to convert InputText to PayloadText.

    oneOf

    string

    use_chat_templatebooleannullable