Class SimpleToken

Represents a generic, abstract token, which is a sequence of characters in the input. A token is identified using a tokenizer, which breaks up the sequence of characters in the input into a sequence of tokens. That token sequence is non-overlapping, but not necessarily contiguous.

Inheritance
object
SegmentElement
Token
SimpleToken
GenericPlaceableToken
Inherited Members
Namespace: SdlSdl.LanguagePlatformCoreTokenization
Assembly: Sdl.LanguagePlatform.Core.dll
Syntax
[DataContract]
public class SimpleToken : Token

Constructors

SimpleToken()

Initializes a new instance with default values.

Declaration
public SimpleToken()

SimpleToken(SimpleToken)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration
public SimpleToken(SimpleToken other)
Parameters
Type Name Description
SimpleToken other

The instance to create a copy for

SimpleToken(string)

Initializes a new instance with the specified token text, and a token type of Word.

Declaration
public SimpleToken(string text)
Parameters
Type Name Description
string text

SimpleToken(string, TokenType)

Initializes a new instance with the specified values.

Declaration
public SimpleToken(string text, TokenType t)
Parameters
Type Name Description
string text

The token text
TokenType t

The token type

Properties

IsPlaceable

IsPlaceable. Returns true if the token represents an acronym, a variable, a URI, or a generic text placeable.

Declaration
public override bool IsPlaceable { get; }
Property Value
Type Description
bool
Overrides

IsStopword

Gets or sets whether this token is a stop word

Declaration
public bool IsStopword { get; set; }
Property Value
Type Description
bool

IsSubstitutable

IsSubstitutable. Returns true if this token represents a variable.

Declaration
public override bool IsSubstitutable { get; }
Property Value
Type Description
bool
Overrides

Stem

An optional stem to assign to this token. Stems are used for TM search and scoring.

Declaration
[DataMember]
public string Stem { get; set; }
Property Value
Type Description
string

Methods

AcceptSegmentElementVisitor(ISegmentElementVisitor)

Visitor pattern implementation. Calls VisitSimpleToken(SimpleToken).

Declaration
public override void AcceptSegmentElementVisitor(ISegmentElementVisitor visitor)
Parameters
Type Name Description
ISegmentElementVisitor visitor

The visitor instance
Overrides

Duplicate()

See Duplicate()

Declaration
public override SegmentElement Duplicate()
Returns
Type Description
SegmentElement

A new instance that is a deep copy of this instance.
Overrides

Equals(object)

System.Object.Equals(object)
Declaration
public override bool Equals(object obj)
Parameters
Type Name Description
object obj

The object to compare with the current object.
Returns
Type Description
bool

true if the specified object is equal to the current object; otherwise, false.
Overrides

GetHashCode()

System.Object.GetHashCode(object)
Declaration
public override int GetHashCode()
Returns
Type Description
int
Overrides

GetTokenType()

Type
Declaration
protected override TokenType GetTokenType()
Returns
Type Description
TokenType
Overrides

SetTokenType(TokenType)

Overwrites the token type. This should only be done with caution, and to reclassify exchangeable token types (acronyms, words, abbreviations). In particular, do not set the token type to one of the special (localizable) token types.

Declaration
protected override void SetTokenType(TokenType tokenType)
Parameters
Type Name Description
TokenType tokenType
Overrides

ToString()

ToString()
Declaration
public override string ToString()
Returns
Type Description
string

A string representation of the object, for display purposes.
Overrides