10 lines
662 B
Markdown
10 lines
662 B
Markdown
# `MlCommonTokenizationConfig` [interface-MlCommonTokenizationConfig]
|
|
|
|
| Name | Type | Description |
|
|
| - | - | - |
|
|
| `do_lower_case` | boolean | Should the tokenizer lower case the text |
|
|
| `max_sequence_length` | [integer](./integer.md) | Maximum input sequence length for the model |
|
|
| `span` | [integer](./integer.md) | Tokenization spanning options. Special value of -1 indicates no spanning takes place |
|
|
| `truncate` | [MlTokenizationTruncate](./MlTokenizationTruncate.md) | Should tokenization input be automatically truncated before sending to the model for inference |
|
|
| `with_special_tokens` | boolean | Is tokenization completed with special tokens |
|