Tokenizer
@astryxdesign/core v0.7.1 · Tokenizer
Usage
Tokenizer collects search results into removable tokens. The source matches Typeahead: createStaticSource for local data, or a search function for a remote source. Selected items stay out of the menu.
hasCreate turns the current text into a new token. hasClear removes every token. When the input is empty, Backspace removes the last token. Arrow keys move focus between tokens, and Backspace or Delete removes the focused one.
overflow controls what happens when the tokens do not fit:
| Value | Behavior |
|---|---|
none | Tokens wrap and stay visible. |
unfocused-inline | On blur, they collapse to one line ending in +N, then expand in place on focus. |
unfocused-layer | On blur they also collapse to one line. On focus they expand over the field and do not push the content below. |
Width measurement uses useOverflow. The #icon slot sits at the start of the field. #token can replace each selected chip with a custom Token.
Best Practices
| Guideline | Practice |
|---|---|
| Do | Use unfocused-inline when there may be many tags and the field should stay on one line. |
| Do | Turn on hasCreate when people can mint a value, and tell add from create in the change event. |
| Don't | Use Tokenizer for a short fixed list. Use Multi Selector for that. |
Anatomy
| Element | Description |
|---|---|
| Label | The visible text above the input describing what the user is selecting. Also used as the accessible name. |
| Token chips | Removable chips representing each selected item. Each chip shows a label and a remove button. |
| Search input | The text input where users type to search the data source. Hides when maxEntries is reached. |
| Dropdown menu | The search results list that appears below the input as the user types. |
| Spinner | Loading indicator shown at the end of the field while a search is in flight. |
| End content | A trailing slot after the input for action buttons, counts, or other controls. |
| Clear button | A button that removes all selected tokens at once. Shown when clearable is set and tokens are present. |
Examples
Clear
Tokenizer with a built-in clear-all button for bulk removal.
Creatable
Free-text only
Create or search
Free-text tokenizer for creating custom tags, and a combined create-or-search pattern.
End Content
Action button in the end slot
Tokenizer with an action button in the end slot.
Icon
Leading icon reinforces the search affordance
Tokenizer with a leading search icon.
Max Entries
Limited to 3 selections — 1 remaining
Tokenizer with a maximum selection limit. The input stops accepting more once the limit is reached.
Overflow
Inline overflow — content shifts down on expand
Layer overflow — expands as overlay, no layout shift
On blur, tokens collapse to one line. Inline mode pushes content down on expand; layer mode overlays without shifting layout.
States
Disabled, error, warning, and success states.