Search Results for

    Show / Hide Table of Contents

    Class Token

    Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

    Inheritance
    System.Object
    SegmentElement
    Token
    DateTimeToken
    NumberToken
    SimpleToken
    TagToken
    TokenBundle
    Inherited Members
    SegmentElement.Duplicate()
    SegmentElement.GetSimilarity(SegmentElement)
    SegmentElement.GetSimilarity(SegmentElement, Boolean)
    SegmentElement.AcceptSegmentElementVisitor(ISegmentElementVisitor)
    System.Object.ToString()
    System.Object.Equals(System.Object, System.Object)
    System.Object.ReferenceEquals(System.Object, System.Object)
    System.Object.GetType()
    System.Object.MemberwiseClone()
    Namespace: Sdl.LanguagePlatform.Core.Tokenization
    Assembly: Sdl.LanguagePlatform.Core.dll
    Syntax
    public abstract class Token : SegmentElement

    Constructors

    Token()

    Initializes a new instance with default values.

    Declaration
    public Token()

    Token(Token)

    Initializes a new instance with the values from another instance, creating a deep copy.

    Declaration
    public Token(Token other)
    Parameters
    Type Name Description
    Token other

    The instance to copy

    Token(String)

    Initializes a new instance with the specified token text.

    Declaration
    public Token(string text)
    Parameters
    Type Name Description
    System.String text

    The token text

    Properties

    Culture

    The culture of the token

    Declaration
    public CultureInfo Culture { get; set; }
    Property Value
    Type Description
    System.Globalization.CultureInfo

    IsPlaceable

    Gets a boolean value indicating whether this token represents a placeable. Always returns false for this class. Derived classes should override appropriately.

    Declaration
    public virtual bool IsPlaceable { get; }
    Property Value
    Type Description
    System.Boolean

    IsPunctuation

    true iff the token is a general punctuation, an opening punctuation, or a closing punctuation

    Declaration
    public bool IsPunctuation { get; }
    Property Value
    Type Description
    System.Boolean

    IsSubstitutable

    Gets a boolean value indicating whether this token can be substituted (e.g. auto-localized) across segments. All substitutables are placeables, but not all placeables can be substituted (e.g. acronyms).

    Declaration
    public virtual bool IsSubstitutable { get; }
    Property Value
    Type Description
    System.Boolean

    IsWhitespace

    Gets a value which indicates whether this token is a whitespace token or not.

    Declaration
    public bool IsWhitespace { get; }
    Property Value
    Type Description
    System.Boolean

    IsWord

    true iff the token is a word, an abbreviation, or an acronym

    Declaration
    public bool IsWord { get; }
    Property Value
    Type Description
    System.Boolean

    Span

    Gets or sets the span covered by this token in the original segment.

    Declaration
    public SegmentRange Span { get; set; }
    Property Value
    Type Description
    SegmentRange

    Text

    Gets or sets the token's text.

    Declaration
    public virtual string Text { get; set; }
    Property Value
    Type Description
    System.String

    TokenizationContext

    The TokenizationContext in which to apply localization etc.

    Declaration
    public TokenizationContext TokenizationContext { get; set; }
    Property Value
    Type Description
    TokenizationContext

    Type

    Gets, or sets, the TokenType of this token.

    Declaration
    public TokenType Type { get; set; }
    Property Value
    Type Description
    TokenType

    Methods

    Equals(Object)

    System.Object.Equals(object)
    Declaration
    public override bool Equals(object obj)
    Parameters
    Type Name Description
    System.Object obj

    The object to compare with the current object.

    Returns
    Type Description
    System.Boolean

    true if the specified object is equal to the current object; otherwise, false.

    Overrides
    System.Object.Equals(System.Object)

    GetBundleSimilarity(SegmentElement)

    Declaration
    protected SegmentElement.Similarity GetBundleSimilarity(SegmentElement other)
    Parameters
    Type Name Description
    SegmentElement other
    Returns
    Type Description
    SegmentElement.Similarity

    GetHashCode()

    System.Object.GetHashCode(object)
    Declaration
    public override int GetHashCode()
    Returns
    Type Description
    System.Int32
    Overrides
    System.Object.GetHashCode()

    GetTokenType()

    Type
    Declaration
    protected abstract TokenType GetTokenType()
    Returns
    Type Description
    TokenType

    GetWeakHashCode()

    See GetWeakHashCode()

    Declaration
    public override int GetWeakHashCode()
    Returns
    Type Description
    System.Int32

    A hash code

    Overrides
    SegmentElement.GetWeakHashCode()

    SetTokenType(TokenType)

    Type
    Declaration
    protected virtual void SetTokenType(TokenType tokenType)
    Parameters
    Type Name Description
    TokenType tokenType

    On this page

    • Constructors
      • Token()
      • Token(Token)
      • Token(String)
    • Properties
      • Culture
      • IsPlaceable
      • IsPunctuation
      • IsSubstitutable
      • IsWhitespace
      • IsWord
      • Span
      • Text
      • TokenizationContext
      • Type
    • Methods
      • Equals(Object)
      • GetBundleSimilarity(SegmentElement)
      • GetHashCode()
      • GetTokenType()
      • GetWeakHashCode()
      • SetTokenType(TokenType)
    Back to top Generated by DocFX