Class CharSequenceToken
A Token representing sequence of characters.
Implements
Inherited Members
Namespace: Sdl.Core.Bcm.BcmModel.Tokenization.Tokens
Assembly: Sdl.Core.Bcm.BcmModel.dll
Syntax
[DataContract]
public class CharSequenceToken : Token, ICloneable<Token>Remarks
This is used in contexts (such as tokenization of some far-east languages) where word boundaries cannot easily be determined.
Constructors
CharSequenceToken()
Initializes a new instance of the CharSequenceToken class.
Declaration
public CharSequenceToken()CharSequenceToken(string)
Initializes a new instance of the CharSequenceToken class.
Declaration
public CharSequenceToken(string text)Parameters
| Type | Name | Description |
|---|---|---|
| string | text | The token text |
Properties
Type
Gets the type of Token.
Declaration
[DataMember(Name = "type")]
protected override string Type { get; }Property Value
| Type | Description |
|---|---|
| string | The type. |