Class GenericPlaceableToken

Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

Inheritance

object

SegmentElement

Token

SimpleToken

GenericPlaceableToken

Inherited Members

SimpleToken.SetTokenType(TokenType)

SimpleToken.GetTokenType()

SimpleToken.ToString()

SimpleToken.AcceptSegmentElementVisitor(ISegmentElementVisitor)

SimpleToken.Stem

SimpleToken.IsStopword

Token.GetWeakHashCode()

Token.GetBundleSimilarity(SegmentElement)

Token.GetIdentityStringAsync()

Token.TokenizationContext

Token.Text

Token.Span

Token.Type

Token.IsWord

Token.IsPunctuation

Token.IsWhitespace

Token.Culture

SegmentElement.GetSimilarity(SegmentElement)

SegmentElement.GetSimilarity(SegmentElement, bool)

SegmentElement.GetSimilarityAsync(SegmentElement)

SegmentElement.GetSimilarityAsync(SegmentElement, bool)

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetType()

object.MemberwiseClone()

Namespace: Sdl.LanguagePlatform.Core.Tokenization

Assembly: Sdl.LanguagePlatform.Core.dll

Syntax

[DataContract]
public class GenericPlaceableToken : SimpleToken

Constructors

GenericPlaceableToken(GenericPlaceableToken)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration

public GenericPlaceableToken(GenericPlaceableToken other)

Parameters

Type	Name	Description
GenericPlaceableToken	other	The instance to create a copy for

GenericPlaceableToken(string, string, bool)

Initializes a new instance with the specified token text, token class, and substitution flag. The token type will be set to OtherTextPlaceable.

Declaration

public GenericPlaceableToken(string text, string tokenClass, bool isSubstitutable)

Parameters

Type	Name	Description
string	text
string	tokenClass
bool	isSubstitutable

Properties

IsPlaceable

IsPlaceable. Always returns true for instances of this class.

Declaration

public override bool IsPlaceable { get; }

Property Value

Type	Description
bool

Overrides

SimpleToken.IsPlaceable

IsSubstitutable

IsSubstitutable. Returns the value of the corresponding constructor flag.

Declaration

public override bool IsSubstitutable { get; }

Property Value

Type	Description
bool

Overrides

SimpleToken.IsSubstitutable

TokenClass

Gets the value of the token class which has been used during construction.

Declaration

public string TokenClass { get; }

Property Value

Type	Description
string

Methods

Duplicate()

See Duplicate()

Declaration

public override SegmentElement Duplicate()

Returns

Type	Description
SegmentElement	A new instance that is a deep copy of this instance.

Overrides

SimpleToken.Duplicate()

Equals(object)

System.Object.Equals(object)

Declaration

public override bool Equals(object obj)

Parameters

Type	Name	Description
object	obj

Returns

Type	Description
bool

Overrides

SimpleToken.Equals(object)

GetHashCode()

System.Object.GetHashCode(object)

Declaration

public override int GetHashCode()

Returns

Type	Description
int

Overrides

SimpleToken.GetHashCode()