Class SimpleToken

Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

Inheritance

object

SegmentElement

Token

SimpleToken

GenericPlaceableToken

Inherited Members

Token.GetWeakHashCode()

Token.GetBundleSimilarity(SegmentElement)

Token.GetIdentityStringAsync()

Token.TokenizationContext

Token.Text

Token.Span

Token.Type

Token.IsWord

Token.IsPunctuation

Token.IsWhitespace

Token.Culture

SegmentElement.GetSimilarity(SegmentElement)

SegmentElement.GetSimilarity(SegmentElement, bool)

SegmentElement.GetSimilarityAsync(SegmentElement)

SegmentElement.GetSimilarityAsync(SegmentElement, bool)

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetType()

object.MemberwiseClone()

Namespace: Sdl.LanguagePlatform.Core.Tokenization

Assembly: Sdl.LanguagePlatform.Core.dll

Syntax

[DataContract]
public class SimpleToken : Token

Constructors

SimpleToken()

Initializes a new instance with default values.

Declaration

public SimpleToken()

SimpleToken(SimpleToken)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration

public SimpleToken(SimpleToken other)

Parameters

Type	Name	Description
SimpleToken	other	The instance to create a copy for

SimpleToken(string)

Initializes a new instance with the specified token text, and a token type of Word.

Declaration

public SimpleToken(string text)

Parameters

Type	Name	Description
string	text

SimpleToken(string, TokenType)

Initializes a new instance with the specified values.

Declaration

public SimpleToken(string text, TokenType t)

Parameters

Type	Name	Description
string	text	The token text
TokenType	t	The token type

Properties

IsPlaceable

IsPlaceable. Returns true if the token represents an acronym, a variable, a URI, or a generic text placeable.

Declaration

public override bool IsPlaceable { get; }

Property Value

Type	Description
bool

Overrides

Token.IsPlaceable

IsStopword

Gets or sets whether this token is a stop word

Declaration

public bool IsStopword { get; set; }

Property Value

Type	Description
bool

IsSubstitutable

IsSubstitutable. Returns true if this token represents a variable.

Declaration

public override bool IsSubstitutable { get; }

Property Value

Type	Description
bool

Overrides

Token.IsSubstitutable

Stem

An optional stem to assign to this token. Stems are used for TM search and scoring.

Declaration

[DataMember]
public string Stem { get; set; }

Property Value

Type	Description
string

Methods

AcceptSegmentElementVisitor(ISegmentElementVisitor)

Visitor pattern implementation. Calls VisitSimpleToken(SimpleToken).

Declaration

public override void AcceptSegmentElementVisitor(ISegmentElementVisitor visitor)

Parameters

Type	Name	Description
ISegmentElementVisitor	visitor	The visitor instance

Overrides

SegmentElement.AcceptSegmentElementVisitor(ISegmentElementVisitor)

Duplicate()

See Duplicate()

Declaration

public override SegmentElement Duplicate()

Returns

Type	Description
SegmentElement	A new instance that is a deep copy of this instance.

Overrides

SegmentElement.Duplicate()

Equals(object)

System.Object.Equals(object)

Declaration

public override bool Equals(object obj)

Parameters

Type	Name	Description
object	obj	The object to compare with the current object.

Returns

Type	Description
bool	true if the specified object is equal to the current object; otherwise, false.

Overrides

Token.Equals(object)

GetHashCode()

System.Object.GetHashCode(object)

Declaration

public override int GetHashCode()

Returns

Type	Description
int

Overrides

Token.GetHashCode()

GetTokenType()

Type

Declaration

protected override TokenType GetTokenType()

Returns

Type	Description
TokenType

Overrides

Token.GetTokenType()

SetTokenType(TokenType)

Overwrites the token type. This should only be done with caution, and to reclassify exchangeable token types (acronyms, words, abbreviations). In particular, do not set the token type to one of the special (localizable) token types.

Declaration

protected override void SetTokenType(TokenType tokenType)

Parameters

Type	Name	Description
TokenType	tokenType

Overrides

Token.SetTokenType(TokenType)

ToString()

ToString()

Declaration

public override string ToString()

Returns

Type	Description
string	A string representation of the object, for display purposes.

Overrides

object.ToString()