Class CharSequenceToken

A Token representing sequence of characters.

Inheritance

System.Object

Token

CharSequenceToken

Implements

ICloneable<Token>

Inherited Members

Token.Clone()

Token.ToString()

Token.Text

Token.Span

Namespace: Sdl.Core.Bcm.BcmModel.Tokenization.Tokens

Assembly: Sdl.Core.Bcm.BcmModel.dll

Syntax

public class CharSequenceToken : Token, ICloneable<Token>

Remarks

This is used in contexts (such as tokenization of some far-east languages) where word boundaries cannot easily be determined.

Constructors

CharSequenceToken()

Initializes a new instance of the CharSequenceToken class.

Declaration

public CharSequenceToken()

CharSequenceToken(String)

Initializes a new instance of the CharSequenceToken class.

Declaration

public CharSequenceToken(string text)

Parameters

Type	Name	Description
System.String	text	The token text

Properties

Type

Gets the type of Token.

Declaration

protected override string Type { get; }

Property Value

Type	Description
System.String	The type.

Overrides

Token.Type

Implements

ICloneable<T>

See Also

Token