Class Token
Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.
Inherited Members
Namespace: Sdl.LanguagePlatform.Core.Tokenization
Assembly: Sdl.LanguagePlatform.Core.dll
Syntax
public abstract class Token : SegmentElement
Constructors
Token()
Initializes a new instance with default values.
Declaration
public Token()
Token(Token)
Initializes a new instance with the values from another instance, creating a deep copy.
Declaration
public Token(Token other)
Parameters
Type | Name | Description |
---|---|---|
Token | other | The instance to copy |
Token(String)
Initializes a new instance with the specified token text.
Declaration
public Token(string text)
Parameters
Type | Name | Description |
---|---|---|
String | text | The token text |
Properties
Culture
The culture of the token
Declaration
public CultureCode Culture { get; set; }
Property Value
Type | Description |
---|---|
CultureCode |
IsPlaceable
Gets a boolean value indicating whether this token represents a placeable.
Always returns false
for this class. Derived
classes should override appropriately.
Declaration
public virtual bool IsPlaceable { get; }
Property Value
Type | Description |
---|---|
Boolean |
IsPunctuation
true iff the token is a general punctuation, an opening punctuation, or a closing punctuation
Declaration
public bool IsPunctuation { get; }
Property Value
Type | Description |
---|---|
Boolean |
IsSubstitutable
Gets a boolean value indicating whether this token can be substituted (e.g. auto-localized) across segments. All substitutables are placeables, but not all placeables can be substituted (e.g. acronyms).
Declaration
public virtual bool IsSubstitutable { get; }
Property Value
Type | Description |
---|---|
Boolean |
IsWhitespace
Gets a value which indicates whether this token is a whitespace token or not.
Declaration
public bool IsWhitespace { get; }
Property Value
Type | Description |
---|---|
Boolean |
IsWord
true iff the token is a word, an abbreviation, or an acronym
Declaration
public bool IsWord { get; }
Property Value
Type | Description |
---|---|
Boolean |
Span
Gets or sets the span covered by this token in the original segment.
Declaration
public SegmentRange Span { get; set; }
Property Value
Type | Description |
---|---|
SegmentRange |
Text
Gets or sets the token's text.
Declaration
public virtual string Text { get; set; }
Property Value
Type | Description |
---|---|
String |
TokenizationContext
The TokenizationContext in which to apply localization etc.
Declaration
public TokenizationContext TokenizationContext { get; set; }
Property Value
Type | Description |
---|---|
TokenizationContext |
Type
Gets, or sets, the TokenType of this token.
Declaration
public TokenType Type { get; set; }
Property Value
Type | Description |
---|---|
TokenType |
Methods
Equals(Object)
System.Object.Equals(object)
Declaration
public override bool Equals(object obj)
Parameters
Type | Name | Description |
---|---|---|
Object | obj | The object to compare with the current object. |
Returns
Type | Description |
---|---|
Boolean | true if the specified object is equal to the current object; otherwise, false. |
Overrides
GetBundleSimilarity(SegmentElement)
Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.
Declaration
protected SegmentElement.Similarity GetBundleSimilarity(SegmentElement other)
Parameters
Type | Name | Description |
---|---|---|
SegmentElement | other |
Returns
Type | Description |
---|---|
SegmentElement.Similarity |
GetHashCode()
System.Object.GetHashCode(object)
Declaration
public override int GetHashCode()
Returns
Type | Description |
---|---|
Int32 |
Overrides
GetIdentityStringAsync()
See Sdl.LanguagePlatform.Core.Tokenization.IToken.GetIdentityStringAsync
Declaration
public Task<string> GetIdentityStringAsync()
Returns
Type | Description |
---|---|
Task<String> |
GetTokenType()
Declaration
protected abstract TokenType GetTokenType()
Returns
Type | Description |
---|---|
TokenType |
GetWeakHashCode()
Declaration
public override int GetWeakHashCode()
Returns
Type | Description |
---|---|
Int32 | A hash code |
Overrides
SetTokenType(TokenType)
Declaration
protected virtual void SetTokenType(TokenType tokenType)
Parameters
Type | Name | Description |
---|---|---|
TokenType | tokenType |