# Tokenizer

## Usage

`Tokenizer` collects search results into removable tokens. The source matches Typeahead: `createStaticSource` for local data, or a `search` function for a remote source. Selected items stay out of the menu.

`hasCreate` turns the current text into a new token. `hasClear` removes every token. When the input is empty, `Backspace` removes the last token. Arrow keys move focus between tokens, and `Backspace` or `Delete` removes the focused one.

`overflow` controls what happens when the tokens do not fit:

| Value | Behavior |
| --- | --- |
| `none` | Tokens wrap and stay visible. |
| `unfocused-inline` | On blur, they collapse to one line ending in `+N`, then expand in place on focus. |
| `unfocused-layer` | On blur they also collapse to one line. On focus they expand over the field and do not push the content below. |

Width measurement uses `useOverflow`. The `#icon` slot sits at the start of the field. `#token` can replace each selected chip with a custom `Token`.

## Best Practices

| Guideline | Practice |
| --- | --- |
| Do | Use `unfocused-inline` when there may be many tags and the field should stay on one line. |
| Do | Turn on `hasCreate` when people can mint a value, and tell `add` from `create` in the change event. |
| Don't | Use Tokenizer for a short fixed list. Use Multi Selector for that. |

### Anatomy

- **Label**: The visible text above the input describing what the user is selecting. Also used as the accessible name.
- **Token chips**: Removable chips representing each selected item. Each chip shows a label and a remove button.
- **Search input**: The text input where users type to search the data source. Hides when maxEntries is reached.
- **Dropdown menu**: The search results list that appears below the input as the user types.
- **Spinner**: Loading indicator shown at the end of the field while a search is in flight.
- **End content**: A trailing slot after the input for action buttons, counts, or other controls.
- **Clear button**: A button that removes all selected tokens at once. Shown when clearable is set and tokens are present.

## Examples

### Clear

Tokenizer with a built-in clear-all button for bulk removal.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const people = computed(() => [
  { id: 'alice', label: 'Alice Johnson' },
  { id: 'bob', label: 'Bob Smith' },
  { id: 'charlie', label: 'Charlie Brown' },
  { id: 'diana', label: 'Diana Prince' },
  { id: 'eve', label: 'Eve Williams' },
])
const source = computed(() => createStaticSource(people.value))
const selected = ref<SearchableItem[]>([])
selected.value = people.value.slice(0, 2)
</script>

<template>
  <div class="demo-container">
    <Tokenizer
      v-model="selected"
      :search-source="source"
      label="Team Members"
      placeholder="Search people..."
      :debounce-ms="0"
      has-clear
    />
  </div>
</template>

<style scoped>
.demo-container {
  width: 20rem;
  max-width: calc(100vw - 4rem);
}
</style>
```

### Creatable

Free-text tokenizer for creating custom tags, and a combined create-or-search pattern.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const tags = ref<SearchableItem[]>([])
const members = ref<SearchableItem[]>([])
const emptySource = createStaticSource<SearchableItem>([])
const people = computed(() => createStaticSource([
  { id: 'alice', label: 'Alice Johnson' },
  { id: 'bob', label: 'Bob Smith' },
  { id: 'charlie', label: 'Charlie Brown' },
  { id: 'diana', label: 'Diana Prince' },
  { id: 'eve', label: 'Eve Williams' },
]))
</script>

<template>
  <div class="demo-stack">
    <p class="demo-caption">
      Free-text only
    </p>
    <Tokenizer
      v-model="tags"
      :search-source="emptySource"
      label="Tags"
      placeholder="Type a tag and press Enter..."
      has-create
      :debounce-ms="0"
    />
    <p class="demo-caption">
      Create or search
    </p>
    <Tokenizer
      v-model="members"
      :search-source="people"
      label="Team Members"
      placeholder="Search or type a new name..."
      has-create
      has-entries-on-focus
      :debounce-ms="0"
    />
  </div>
</template>

<style scoped>
.demo-stack {
  display: flex;
  flex-direction: column;
  gap: 0.75rem;
  width: 25rem;
  max-width: calc(100vw - 4rem);
}
.demo-caption {
  margin: 0;
  font-size: 0.875rem;
  color: var(--vp-c-text-2);
}
</style>
```

### End Content

Tokenizer with an action button in the end slot.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Button } from '@astryx-vue/core/Button'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const people = computed(() => [
  { id: 'alice', label: 'Alice Johnson' },
  { id: 'bob', label: 'Bob Smith' },
  { id: 'charlie', label: 'Charlie Brown' },
  { id: 'diana', label: 'Diana Prince' },
  { id: 'eve', label: 'Eve Williams' },
])
const source = computed(() => createStaticSource(people.value))
const selected = ref<SearchableItem[]>([])
selected.value = [people.value[0]!, people.value[2]!]
</script>

<template>
  <div class="demo-stack">
    <p class="demo-caption">
      Action button in the end slot
    </p>
    <Tokenizer
      v-model="selected"
      :search-source="source"
      label="Team Members"
      placeholder="Search people..."
      :debounce-ms="0"
    >
      <template #end>
        <Button label="Apply" variant="primary" size="sm" />
      </template>
    </Tokenizer>
  </div>
</template>

<style scoped>
.demo-stack {
  display: flex;
  flex-direction: column;
  gap: 0.5rem;
  width: 25rem;
  max-width: calc(100vw - 4rem);
}
.demo-caption {
  margin: 0;
  font-size: 0.875rem;
  color: var(--vp-c-text-2);
}
</style>
```

### Icon

Tokenizer with a leading search icon.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const people = computed(() => [
  { id: 'alice', label: 'Alice Johnson' },
  { id: 'bob', label: 'Bob Smith' },
  { id: 'charlie', label: 'Charlie Brown' },
  { id: 'diana', label: 'Diana Prince' },
  { id: 'eve', label: 'Eve Williams' },
])
const source = computed(() => createStaticSource(people.value))
const selected = ref<SearchableItem[]>([])
selected.value = [people.value[0]!, people.value[2]!]
</script>

<template>
  <div class="demo-stack">
    <p class="demo-caption">
      Leading icon reinforces the search affordance
    </p>
    <Tokenizer
      v-model="selected"
      :search-source="source"
      label="Team Members"
      placeholder="Search people..."
      :debounce-ms="0"
    >
      <template #icon>
        <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><path stroke-linecap="round" stroke-linejoin="round" d="m21 21-5.197-5.197m0 0A7.5 7.5 0 1 0 5.196 5.196a7.5 7.5 0 0 0 10.607 10.607Z" /></svg>
      </template>
    </Tokenizer>
  </div>
</template>

<style scoped>
.demo-stack {
  display: flex;
  flex-direction: column;
  gap: 0.5rem;
  width: 25rem;
  max-width: calc(100vw - 4rem);
}
.demo-caption {
  margin: 0;
  font-size: 0.875rem;
  color: var(--vp-c-text-2);
}
.demo-stack :deep(svg) {
  width: 1rem;
  height: 1rem;
}
</style>
```

### Max Entries

Tokenizer with a maximum selection limit. The input stops accepting more once the limit is reached.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const maxSkills = 3
const skills = computed(() => [
  { id: 'react', label: 'React' },
  { id: 'ts', label: 'TypeScript' },
  { id: 'graphql', label: 'GraphQL' },
  { id: 'node', label: 'Node.js' },
  { id: 'python', label: 'Python' },
])
const source = computed(() => createStaticSource(skills.value))
const selected = ref<SearchableItem[]>([skills.value[0]!, skills.value[1]!])
</script>

<template>
  <div class="demo-stack">
    <p class="demo-caption">
      {{ `Limited to ${maxSkills} selections — ${maxSkills - selected.length} remaining` }}
    </p>
    <Tokenizer
      v-model="selected"
      :search-source="source"
      label="Top Skills"
      placeholder="Search skills..."
      :description="`Choose up to ${maxSkills} skills`"
      :max-entries="maxSkills"
      :debounce-ms="0"
    />
  </div>
</template>

<style scoped>
.demo-stack {
  display: flex;
  flex-direction: column;
  gap: 0.5rem;
  width: 25rem;
  max-width: calc(100vw - 4rem);
}
.demo-caption {
  margin: 0;
  font-size: 0.875rem;
  color: var(--vp-c-text-2);
}
</style>
```

### Overflow

On blur, tokens collapse to one line. Inline mode pushes content down on expand; layer mode overlays without shifting layout.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const people = computed<SearchableItem[]>(() => [
  { id: 'alice', label: 'Alice Johnson' },
  { id: 'bob', label: 'Bob Smith' },
  { id: 'charlie', label: 'Charlie Brown' },
  { id: 'diana', label: 'Diana Prince' },
  { id: 'eve', label: 'Eve Williams' },
  { id: 'frank', label: 'Frank Miller' },
])
const source = computed(() => createStaticSource(people.value))
const inlineValue = ref<SearchableItem[]>([])
const layerValue = ref<SearchableItem[]>([])
inlineValue.value = people.value.map(item => ({ ...item }))
layerValue.value = people.value.map(item => ({ ...item }))
</script>

<template>
  <div class="demo-stack">
    <p class="demo-caption">
      Inline overflow — content shifts down on expand
    </p>
    <Tokenizer
      v-model="inlineValue"
      :search-source="source"
      label="Inline Overflow"
      placeholder="Add more..."
      overflow="unfocused-inline"
      :debounce-ms="0"
    />
    <p class="demo-caption">
      Layer overflow — expands as overlay, no layout shift
    </p>
    <Tokenizer
      v-model="layerValue"
      :search-source="source"
      label="Layer Overflow"
      placeholder="Add more..."
      overflow="unfocused-layer"
      :debounce-ms="0"
    />
  </div>
</template>

<style scoped>
.demo-stack {
  display: flex;
  flex-direction: column;
  gap: 0.75rem;
  width: 25rem;
  max-width: calc(100vw - 4rem);
}
.demo-caption {
  margin: 0;
  font-size: 0.875rem;
  color: var(--vp-c-text-2);
}
</style>
```

### States

Disabled, error, warning, and success states.

```vue
<script setup lang="ts">
import type { SearchableItem } from '@astryx-vue/core'
import { createStaticSource } from '@astryx-vue/core'
import { Tokenizer } from '@astryx-vue/core/Tokenizer'
import { computed, ref } from 'vue'

const people = computed(() => [
  { id: 'alice', label: 'Alice Johnson' },
  { id: 'bob', label: 'Bob Smith' },
  { id: 'charlie', label: 'Charlie Brown' },
  { id: 'diana', label: 'Diana Prince' },
])
const source = computed(() => createStaticSource(people.value))
const errorValue = ref<SearchableItem[]>([])
const warningValue = ref<SearchableItem[]>([])
const successValue = ref<SearchableItem[]>([])
warningValue.value = [people.value[0]!]
successValue.value = [people.value[1]!, people.value[3]!]
const disabledValue = computed(() => [people.value[0]!, people.value[2]!])
</script>

<template>
  <div class="demo-stack">
    <Tokenizer
      :model-value="disabledValue"
      :search-source="source"
      label="Disabled field"
      disabled
    />
    <Tokenizer
      v-model="errorValue"
      :search-source="source"
      label="Error message"
      placeholder="Search people..."
      required
      status="error"
      status-message="At least one reviewer is required"
      :debounce-ms="0"
    />
    <Tokenizer
      v-model="warningValue"
      :search-source="source"
      label="Warning message"
      placeholder="Search people..."
      status="warning"
      status-message="Consider adding at least 2 approvers"
      :debounce-ms="0"
    />
    <Tokenizer
      v-model="successValue"
      :search-source="source"
      label="Success message"
      placeholder="Search people..."
      status="success"
      status-message="All required reviewers added"
      :debounce-ms="0"
    />
  </div>
</template>

<style scoped>
.demo-stack {
  display: flex;
  flex-direction: column;
  gap: 1rem;
  width: 25rem;
  max-width: calc(100vw - 4rem);
}
</style>
```

## Tokenizer Props

| Name | Type | Default | Description |
| --- | --- | --- | --- |
| `modelValue` | `T[]` | `[]` | Selected items. |
| `searchSource` | `SearchSource` | Required | Static or async source. |
| `label` | `string` | Required | Field label. |
| `description` | `string` | — | Helper text under the label. |
| `labelTooltip` | `string` | — | Tooltip text on an info icon next to the label. |
| `isLabelHidden` | `boolean` | `false` | Hides the label visually while keeping it accessible. `labelHidden` is the older name; either one hides it. |
| `containerClass` / `controlClass` | `class` | — | Field shell and input box. A root class lands on the box, not the native input. Item actions behave like Typeahead. |
| `placeholder` | `string` | `Add…` / `添加…` | Input placeholder. Shown only while no tokens are selected. |
| `hasCreate` | `boolean` | `false` | Let users turn free text into a new token. A "Create …" row appears for text that matches nothing else, and the change type is `create`. |
| `creatable` | `boolean` | `false` | Legacy name for `hasCreate`. Either one turns it on. |
| `overflow` | `'none' \| 'unfocused-inline' \| 'unfocused-layer'` | `'none'` | Whether blur collapses the tokens to one line with `+N`. |
| `maxEntries` | `number` | — | Maximum number of tokens. The input becomes read-only at the limit. |
| `debounceMs` | `number` | `150` | Wait after typing before searching. |
| `minQueryLength` | `number` | `1` | Shortest query that gets searched, counted in visible characters. Below it nothing is searched and the menu stays closed — unless `hasCreate` is on, because a "Create …" row comes from the typed text and costs no search. |
| `hasEntriesOnFocus` | `boolean` | `false` | Show results on focus before the user types. |
| `showOnFocus` | `boolean` | `false` | Legacy name for `hasEntriesOnFocus`. Either one turns it on. |
| `hasAutoFocus` | `boolean` | `false` | Focus the input on mount. |
| `menuWidth` | `number` | — | Requested dropdown width in pixels. Never narrower than the field, and clamped to the viewport. |
| `maxMenuItems` | `number` | `10` | Maximum number of results in the menu. A "Create …" row is offered on top of them. |
| `hasClear` | `boolean` | `false` | Show a clear-all button that removes every token. |
| `clearable` | `boolean` | `false` | Legacy name for `hasClear`. Either one turns it on. |
| `emptySearchText` | `string \| null` | `'No results found'` / `'未找到结果'` | Text shown when the query matched nothing. `null` means "not given" and falls through to the default; an empty string renders nothing. For rich content use the `empty` slot. |
| `emptyText` | `string \| null` | — | Legacy name for `emptySearchText`. |
| `disabled` | `boolean` | `false` | Disable the field and every token interaction. |
| `disabledMessage` | `string` | — | Explains why the field is disabled. With `disabled`, the input stays focusable through `aria-disabled` and `readonly`, and the message shows as a tooltip on hover or keyboard focus. |
| `disabledReason` | `string` | — | Legacy name for `disabledMessage`. |
| `status` | `'default' \| 'success' \| 'warning' \| 'error'` | `'default'` | Status style. |
| `statusMessage` | `string` | — | Status text under the field. |
| `size` | `'sm' \| 'md' \| 'lg'` | `'md'` | Input and token size. |
| `required` | `boolean` | `false` | Marks the field as required. |
| `optional` | `boolean` | `false` | Shows an optional indicator on the label. |
| `name` | `string` | — | Form field name. Each selected id is submitted once. |
| `width` | `number \| string` | — | Width of the whole field (number = pixels, string used as-is). |

## Tokenizer Emits

| Name | Parameters | Description |
| --- | --- | --- |
| `update:modelValue` | `value: T[]` | Emitted when the selected items change. |
| `change` | `value: T[]`, `change: TokenizerChange` | Includes the change type: `add`, `create`, `remove`, or `clear`. |
| `changeQuery` | `query: string` | Emitted when the query text changes. |
| `search` | `query: string` | Legacy name for `changeQuery`. Both fire. |
| `openChange` | `isOpen: boolean` | Emitted when the menu opens or closes. |
| `focus` | `event: FocusEvent` | Emitted when focus enters the field from outside. |
| `blur` | `event: FocusEvent` | Emitted when focus leaves the whole field, not just the input. |

## Tokenizer Slots

| Name | Parameters | Description |
| --- | --- | --- |
| `icon` | — | Icon at the start of the field, before any token. |
| `end` | — | Content at the end of the field, such as a button or a result count. |
| `token` | `{ item, remove }` | Custom selected token. Replaces the default `Token`. |
| `item` | `{ item, isActive, isSelected }` | Custom menu row. Replaces the default `TypeaheadItem`. |
| `empty` | — | Custom empty state. |

Native attributes and listeners that are not listed above land on the input element, so `@keydown`, `@paste` and `aria-*` work as they would on a plain `<input>`. A `keydown` listener runs before the internal token navigation; call `event.preventDefault()` to skip it.