Class Token

Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

Inheritance

object

SegmentElement

Token

DateTimeToken

NumberToken

SimpleToken

TagToken

TokenBundle

Inherited Members

SegmentElement.Duplicate()

SegmentElement.GetSimilarity(SegmentElement)

SegmentElement.GetSimilarity(SegmentElement, bool)

SegmentElement.GetSimilarityAsync(SegmentElement)

SegmentElement.GetSimilarityAsync(SegmentElement, bool)

SegmentElement.AcceptSegmentElementVisitor(ISegmentElementVisitor)

object.ToString()

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetType()

object.MemberwiseClone()

Namespace: SdlSdl.LanguagePlatformCoreTokenization

Assembly: Sdl.LanguagePlatform.Core.dll

Syntax

[DataContract]
[KnownType(typeof(DateTimeToken))]
[KnownType(typeof(NumberToken))]
[KnownType(typeof(MeasureToken))]
[KnownType(typeof(SimpleToken))]
[KnownType(typeof(GenericPlaceableToken))]
[KnownType(typeof(TagToken))]
public abstract class Token : SegmentElement

Constructors

Token()

Initializes a new instance with default values.

Declaration

public Token()

Token(Token)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration

public Token(Token other)

Parameters

Type	Name	Description
Token	other	The instance to copy

Token(string)

Initializes a new instance with the specified token text.

Declaration

public Token(string text)

Parameters

Type	Name	Description
string	text	The token text

Properties

Culture

The culture of the token

Declaration

public CultureCode Culture { get; set; }

Property Value

Type	Description
CultureCode

IsPlaceable

Gets a boolean value indicating whether this token represents a placeable. Always returns false for this class. Derived classes should override appropriately.

Declaration

public virtual bool IsPlaceable { get; }

Property Value

Type	Description
bool

IsPunctuation

true iff the token is a general punctuation, an opening punctuation, or a closing punctuation

Declaration

public bool IsPunctuation { get; }

Property Value

Type	Description
bool

IsSubstitutable

Gets a boolean value indicating whether this token can be substituted (e.g. auto-localized) across segments. All substitutables are placeables, but not all placeables can be substituted (e.g. acronyms).

Declaration

public virtual bool IsSubstitutable { get; }

Property Value

Type	Description
bool

IsWhitespace

Gets a value which indicates whether this token is a whitespace token or not.

Declaration

public bool IsWhitespace { get; }

Property Value

Type	Description
bool

IsWord

true iff the token is a word, an abbreviation, or an acronym

Declaration

public bool IsWord { get; }

Property Value

Type	Description
bool

Span

Gets or sets the span covered by this token in the original segment.

Declaration

[DataMember]
public SegmentRange Span { get; set; }

Property Value

Type	Description
SegmentRange

Text

Gets or sets the token's text.

Declaration

[DataMember]
public virtual string Text { get; set; }

Property Value

Type	Description
string

TokenizationContext

The TokenizationContext in which to apply localization etc.

Declaration

public TokenizationContext TokenizationContext { get; set; }

Property Value

Type	Description
TokenizationContext

Type

Gets, or sets, the TokenType of this token.

Declaration

[DataMember]
public TokenType Type { get; set; }

Property Value

Type	Description
TokenType

Methods

Equals(object)

System.Object.Equals(object)

Declaration

public override bool Equals(object obj)

Parameters

Type	Name	Description
object	obj	The object to compare with the current object.

Returns

Type	Description
bool	true if the specified object is equal to the current object; otherwise, false.

Overrides

object.Equals(object)

GetBundleSimilarity(SegmentElement)

Declaration

protected SegmentElement.Similarity GetBundleSimilarity(SegmentElement other)

Parameters

Type	Name	Description
SegmentElement	other

Returns

Type	Description
SegmentElementSimilarity

GetHashCode()

System.Object.GetHashCode(object)

Declaration

public override int GetHashCode()

Returns

Type	Description
int

Overrides

object.GetHashCode()

GetIdentityStringAsync()

See GetIdentityStringAsync()

Declaration

public Task<string> GetIdentityStringAsync()

Returns

Type	Description
Taskstring

GetTokenType()

Type

Declaration

protected abstract TokenType GetTokenType()

Returns

Type	Description
TokenType

GetWeakHashCode()

See GetWeakHashCode()

Declaration

public override int GetWeakHashCode()

Returns

Type	Description
int	A hash code

Overrides

SegmentElement.GetWeakHashCode()

SetTokenType(TokenType)

Type

Declaration

protected virtual void SetTokenType(TokenType tokenType)

Parameters

Type	Name	Description
TokenType	tokenType