Class Token

Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

Inheritance

System.Object

SegmentElement

Token

DateTimeToken

NumberToken

SimpleToken

TagToken

TokenBundle

Inherited Members

SegmentElement.Duplicate()

SegmentElement.GetSimilarity(SegmentElement)

SegmentElement.GetSimilarity(SegmentElement, Boolean)

SegmentElement.GetSimilarityAsync(SegmentElement)

SegmentElement.GetSimilarityAsync(SegmentElement, Boolean)

SegmentElement.AcceptSegmentElementVisitor(ISegmentElementVisitor)

System.Object.ToString()

System.Object.Equals(System.Object, System.Object)

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.GetType()

System.Object.MemberwiseClone()

Namespace: Sdl.LanguagePlatform.Core.Tokenization

Assembly: Sdl.LanguagePlatform.Core.dll

Syntax

public abstract class Token : SegmentElement

Constructors

Token()

Initializes a new instance with default values.

Declaration

public Token()

Token(Token)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration

public Token(Token other)

Parameters

Type	Name	Description
Token	other	The instance to copy

Token(String)

Initializes a new instance with the specified token text.

Declaration

public Token(string text)

Parameters

Type	Name	Description
System.String	text	The token text

Properties

Culture

The culture of the token

Declaration

public CultureCode Culture { get; set; }

Property Value

Type	Description
Sdl.Core.Globalization.CultureCode

IsPlaceable

Gets a boolean value indicating whether this token represents a placeable. Always returns false for this class. Derived classes should override appropriately.

Declaration

public virtual bool IsPlaceable { get; }

Property Value

Type	Description
System.Boolean

IsPunctuation

true iff the token is a general punctuation, an opening punctuation, or a closing punctuation

Declaration

public bool IsPunctuation { get; }

Property Value

Type	Description
System.Boolean

IsSubstitutable

Gets a boolean value indicating whether this token can be substituted (e.g. auto-localized) across segments. All substitutables are placeables, but not all placeables can be substituted (e.g. acronyms).

Declaration

public virtual bool IsSubstitutable { get; }

Property Value

Type	Description
System.Boolean

IsWhitespace

Gets a value which indicates whether this token is a whitespace token or not.

Declaration

public bool IsWhitespace { get; }

Property Value

Type	Description
System.Boolean

IsWord

true iff the token is a word, an abbreviation, or an acronym

Declaration

public bool IsWord { get; }

Property Value

Type	Description
System.Boolean

Span

Gets or sets the span covered by this token in the original segment.

Declaration

public SegmentRange Span { get; set; }

Property Value

Type	Description
SegmentRange

Text

Gets or sets the token's text.

Declaration

public virtual string Text { get; set; }

Property Value

Type	Description
System.String

TokenizationContext

The TokenizationContext in which to apply localization etc.

Declaration

public TokenizationContext TokenizationContext { get; set; }

Property Value

Type	Description
TokenizationContext

Type

Gets, or sets, the TokenType of this token.

Declaration

public TokenType Type { get; set; }

Property Value

Type	Description
TokenType

Methods

Equals(Object)

System.Object.Equals(object)

Declaration

public override bool Equals(object obj)

Parameters

Type	Name	Description
System.Object	obj	The object to compare with the current object.

Returns

Type	Description
System.Boolean	true if the specified object is equal to the current object; otherwise, false.

Overrides

System.Object.Equals(System.Object)

GetBundleSimilarity(SegmentElement)

Declaration

protected SegmentElement.Similarity GetBundleSimilarity(SegmentElement other)

Parameters

Type	Name	Description
SegmentElement	other

Returns

Type	Description
SegmentElement.Similarity

GetHashCode()

System.Object.GetHashCode(object)

Declaration

public override int GetHashCode()

Returns

Type	Description
System.Int32

Overrides

System.Object.GetHashCode()

GetTokenType()

Type

Declaration

protected abstract TokenType GetTokenType()

Returns

Type	Description
TokenType

GetWeakHashCode()

See GetWeakHashCode()

Declaration

public override int GetWeakHashCode()

Returns

Type	Description
System.Int32	A hash code

Overrides

SegmentElement.GetWeakHashCode()

SetTokenType(TokenType)

Type

Declaration

protected virtual void SetTokenType(TokenType tokenType)

Parameters

Type	Name	Description
TokenType	tokenType