Enum TokenType
The type of a token, e.g. whether the token represents a word, punctuation, etc.
Namespace: Sdl.LanguagePlatform.Core.Tokenization
Assembly: Sdl.LanguagePlatform.Core.dll
Syntax
public enum TokenTypeFields
| Name | Description |
|---|---|
| Abbreviation | The token represents an abbreviation. |
| Acronym | The token represents an acronym. |
| AlphaNumeric | |
| CharSequence | The token represents a character sequence. This is used in contexts (such as tokenization of some far-east languages) where word boundaries cannot easily be determined. |
| ClosingPunctuation | The token represents closing punctuation, such as a right parenthesis, which is typically preceded by the opening counterpart. |
| Date | The token represents a complex date expression. |
| GeneralPunctuation | The token represents general punctuation. |
| Measurement | The token represents a measurement, which is a combination of a number followed by a unit. |
| Number | The token represents a number. |
| OpeningPunctuation | The token represents opening punctuation, such as a left parenthesis, which is typically followed by the closing counterpart. |
| OtherTextPlaceable | The token represents an unspecified text placeable (not a tag placeables), which can be placed, but not auto-substituted. |
| Tag | The token represents a tag. |
| Time | The token represents a complex time expression. |
| Unknown | The token type is unknown or has not yet been determined. |
| Uri | The token represents a URI/URL. |
| UserDefined | The token represents a user-defined entity. |
| Variable | The token represents a substitutable variable. |
| Whitespace | The token represents a non-empty sequence of whitespace. |
| Word | The token represents a word. |