Class SimpleToken

A Token which represents a simple token, such as a word, whitespace, or punctuation.

Inheritance
System.Object
SegmentElement
Token
SimpleToken
GenericPlaceableToken
Inherited Members
System.Object.Equals(System.Object, System.Object)
System.Object.ReferenceEquals(System.Object, System.Object)
System.Object.GetType()
System.Object.MemberwiseClone()
Namespace: Sdl.LanguagePlatform.Core.Tokenization
Assembly: Sdl.LanguagePlatform.Core.dll
Syntax
public class SimpleToken : Token

Constructors

SimpleToken()

Initializes a new instance with default values

Declaration
public SimpleToken()

SimpleToken(SimpleToken)

Initializes a new instance with the values from another instance, creating a deep copy.

Declaration
public SimpleToken(SimpleToken other)
Parameters
Type Name Description
SimpleToken other

The instance to create a copy for

SimpleToken(String)

Initializes a new instance with the specified token text, and a token type of Word.

Declaration
public SimpleToken(string text)
Parameters
Type Name Description
System.String text

SimpleToken(String, TokenType)

Initializes a new instance with the specified values.

Declaration
public SimpleToken(string text, TokenType t)
Parameters
Type Name Description
System.String text

The token text
TokenType t

The token type

Properties

IsPlaceable

IsPlaceable. Returns true if the token represents an acronym, a variable, a URI, or a generic text placeable.

Declaration
public override bool IsPlaceable { get; }
Property Value
Type Description
System.Boolean
Overrides

IsStopword

Gets or sets whether this token is a stop word

Declaration
public bool IsStopword { get; set; }
Property Value
Type Description
System.Boolean

IsSubstitutable

IsSubstitutable. Returns true if this token represents a variable.

Declaration
public override bool IsSubstitutable { get; }
Property Value
Type Description
System.Boolean
Overrides

Stem

An optional stem to assign to this token. Stems are used for TM search and scoring.

Declaration
public string Stem { get; set; }
Property Value
Type Description
System.String

Methods

AcceptSegmentElementVisitor(ISegmentElementVisitor)

Visitor pattern implementation. Calls VisitSimpleToken(SimpleToken).

Declaration
public override void AcceptSegmentElementVisitor(ISegmentElementVisitor visitor)
Parameters
Type Name Description
ISegmentElementVisitor visitor

The visitor instance
Overrides

Duplicate()

See Duplicate()

Declaration
public override SegmentElement Duplicate()
Returns
Type Description
SegmentElement

A new instance that is a deep copy of this instance.
Overrides

Equals(Object)

System.Object.Equals(object)
Declaration
public override bool Equals(object obj)
Parameters
Type Name Description
System.Object obj

The object to compare with the current object.
Returns
Type Description
System.Boolean

true if the specified object is equal to the current object; otherwise, false.
Overrides

GetHashCode()

System.Object.GetHashCode(object)
Declaration
public override int GetHashCode()
Returns
Type Description
System.Int32
Overrides

GetTokenType()

Type
Declaration
protected override TokenType GetTokenType()
Returns
Type Description
TokenType
Overrides

SetTokenType(TokenType)

Overwrites the token type. This should only be done with caution, and to reclassify exchangeable token types (acronyms, words, abbreviations). In particular, do not set the token type to one of the special (localizable) token types.

Declaration
protected override void SetTokenType(TokenType tokenType)
Parameters
Type Name Description
TokenType tokenType
Overrides

ToString()

System.Object.ToString()
Declaration
public override string ToString()
Returns
Type Description
System.String

A string representation of the object, for display purposes.
Overrides
System.Object.ToString()