Class Token

Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

Inheritance
object
SegmentElement
Token
DateTimeToken
NumberToken
SimpleToken
TagToken
TokenBundle
Inherited Members
Namespace: Sdl.LanguagePlatform.Core.Tokenization
Assembly: Sdl.LanguagePlatform.Core.dll
Syntax
[DataContract]
[KnownType(typeof(DateTimeToken))]
[KnownType(typeof(NumberToken))]
[KnownType(typeof(MeasureToken))]
[KnownType(typeof(SimpleToken))]
[KnownType(typeof(GenericPlaceableToken))]
[KnownType(typeof(TagToken))]
public abstract class Token : SegmentElement

Constructors

Token()

Initializes a new instance with default values.

Declaration
public Token()

Token(Token)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration
public Token(Token other)
Parameters
Type Name Description
Token other

The instance to copy

Token(string)

Initializes a new instance with the specified token text.

Declaration
public Token(string text)
Parameters
Type Name Description
string text

The token text

Properties

Culture

The culture of the token

Declaration
public CultureCode Culture { get; set; }
Property Value
Type Description
CultureCode

IsPlaceable

Gets a boolean value indicating whether this token represents a placeable. Always returns false for this class. Derived classes should override appropriately.

Declaration
public virtual bool IsPlaceable { get; }
Property Value
Type Description
bool

IsPunctuation

true iff the token is a general punctuation, an opening punctuation, or a closing punctuation

Declaration
public bool IsPunctuation { get; }
Property Value
Type Description
bool

IsSubstitutable

Gets a boolean value indicating whether this token can be substituted (e.g. auto-localized) across segments. All substitutables are placeables, but not all placeables can be substituted (e.g. acronyms).

Declaration
public virtual bool IsSubstitutable { get; }
Property Value
Type Description
bool

IsWhitespace

Gets a value which indicates whether this token is a whitespace token or not.

Declaration
public bool IsWhitespace { get; }
Property Value
Type Description
bool

IsWord

true iff the token is a word, an abbreviation, or an acronym

Declaration
public bool IsWord { get; }
Property Value
Type Description
bool

Span

Gets or sets the span covered by this token in the original segment.

Declaration
[DataMember]
public SegmentRange Span { get; set; }
Property Value
Type Description
SegmentRange

Text

Gets or sets the token's text.

Declaration
[DataMember]
public virtual string Text { get; set; }
Property Value
Type Description
string

TokenizationContext

The TokenizationContext in which to apply localization etc.

Declaration
public TokenizationContext TokenizationContext { get; set; }
Property Value
Type Description
TokenizationContext

Type

Gets, or sets, the TokenType of this token.

Declaration
[DataMember]
public TokenType Type { get; set; }
Property Value
Type Description
TokenType

Methods

Equals(object)

System.Object.Equals(object)

Declaration
public override bool Equals(object obj)
Parameters
Type Name Description
object obj

The object to compare with the current object.
Returns
Type Description
bool

true if the specified object is equal to the current object; otherwise, false.
Overrides

GetBundleSimilarity(SegmentElement)

Declaration
protected SegmentElement.Similarity GetBundleSimilarity(SegmentElement other)
Parameters
Type Name Description
SegmentElement other
Returns
Type Description
SegmentElement.Similarity

GetHashCode()

System.Object.GetHashCode(object)

Declaration
public override int GetHashCode()
Returns
Type Description
int
Overrides

GetIdentityStringAsync()

See Sdl.LanguagePlatform.Core.Tokenization.IToken.GetIdentityStringAsync()

Declaration
public Task<string> GetIdentityStringAsync()
Returns
Type Description
Task<string>

GetTokenType()

Type

Declaration
protected abstract TokenType GetTokenType()
Returns
Type Description
TokenType

GetWeakHashCode()

See GetWeakHashCode()

Declaration
public override int GetWeakHashCode()
Returns
Type Description
int

A hash code
Overrides

SetTokenType(TokenType)

Type

Declaration
protected virtual void SetTokenType(TokenType tokenType)
Parameters
Type Name Description
TokenType tokenType