Class TokenBundle

A special Token which represents a set of alternatives (i.e. an ambiguous analysis) of other tokens which cover the exactly same input span.

Inheritance

object

SegmentElement

Token

TokenBundle

Implements

IEnumerablePrioritizedToken

IEnumerable

Inherited Members

Token.SetTokenType(TokenType)

Token.Equals(object)

Token.GetHashCode()

Token.GetWeakHashCode()

Token.GetBundleSimilarity(SegmentElement)

Token.GetIdentityStringAsync()

Token.TokenizationContext

Token.Text

Token.Span

Token.Type

Token.IsWord

Token.IsPunctuation

Token.IsWhitespace

Token.Culture

SegmentElement.GetSimilarity(SegmentElement)

SegmentElement.GetSimilarity(SegmentElement, bool)

SegmentElement.GetSimilarityAsync(SegmentElement)

SegmentElement.GetSimilarityAsync(SegmentElement, bool)

object.ToString()

object.Equals(object, object)

object.ReferenceEquals(object, object)

object.GetType()

object.MemberwiseClone()

Namespace: SdlSdl.LanguagePlatformCoreTokenization

Assembly: Sdl.LanguagePlatform.Core.dll

Syntax

public class TokenBundle : Token, IEnumerable<PrioritizedToken>, IEnumerable

Constructors

TokenBundle(Token, int)

Initializes a new instance with the specified values.

Declaration

public TokenBundle(Token t, int priority)

Parameters

Type	Name	Description
Token	t	The first token in the set of alternatives (must not be null). It is also used to set the Text of the bundle.
int	priority	The priority to assign to the first token in the bundle

TokenBundle(IList<PrioritizedToken>)

Initializes a new instance with a list of prioritized tokens.

Declaration

public TokenBundle(IList<PrioritizedToken> items)

Parameters

Type	Name	Description
IListPrioritizedToken	items	The items to add to the list of alternatives. Only a shallow copy is made.

Properties

Count

Gets the number of elements in the collection.

Declaration

public int Count { get; }

Property Value

Type	Description
int

IsPlaceable

IsPlaceable. Is true if any of the alternatives is a placeable token.

Declaration

public override bool IsPlaceable { get; }

Property Value

Type	Description
bool

Overrides

Token.IsPlaceable

IsSubstitutable

IsSubstitutable. Is true if any of the alternatives is a substitutable token.

Declaration

public override bool IsSubstitutable { get; }

Property Value

Type	Description
bool

Overrides

Token.IsSubstitutable

this[int]

Gets the prioritized token at the specified index position. If the index position is invalid, an exception is thrown.

Declaration

public PrioritizedToken this[int index] { get; }

Parameters

Type	Name	Description
int	index	The index

Property Value

Type	Description
PrioritizedToken	The prioritized token at the index position

Methods

AcceptSegmentElementVisitor(ISegmentElementVisitor)

Invalid for this class. Will throw a InvalidOperationException

Declaration

public override void AcceptSegmentElementVisitor(ISegmentElementVisitor visitor)

Parameters

Type	Name	Description
ISegmentElementVisitor	visitor	(ignored)

Overrides

SegmentElement.AcceptSegmentElementVisitor(ISegmentElementVisitor)

Add(Token, int)

Adds a new token to the set of alternatives. Potential duplicates are kept.

Declaration

public void Add(Token t, int priority)

Parameters

Type	Name	Description
Token	t	The token
int	priority	The priority assigned to the token

Add(Token, int, bool)

Adds a new token to the set of alternatives.

Declaration

public void Add(Token t, int priority, bool keepDuplicates)

Parameters

Type	Name	Description
Token	t	The token
int	priority	The priority assigned to the token
bool	keepDuplicates	If true, the token will always be added. If false, and the collection contains a token which Equals t, only the one with the higher priority will be kept.

Contains(Token)

Determines whether the specified token is contained in the set of alternatives.

Declaration

public bool Contains(Token t)

Parameters

Type	Name	Description
Token	t	The token to search

Returns

Type	Description
bool	true if the token is equal to one of the alternatives, and false otherwise

Duplicate()

Invalid for this class. Will throw a InvalidOperationException

Declaration

public override SegmentElement Duplicate()

Returns

Type	Description
SegmentElement

Overrides

SegmentElement.Duplicate()

GetBest()

Obtains the token with the highest priority in the set of alternatives. If two tokens have the same priority, the first one will be the winner.

Declaration

public Token GetBest()

Returns

Type	Description
Token	The token with the highest priority in the set of alternatives.

GetEnumerator()

Gets an enumerator to iterate through the list of alternatives.

Declaration

public IEnumerator<PrioritizedToken> GetEnumerator()

Returns

Type	Description
IEnumeratorPrioritizedToken	An enumerator

GetTokenType()

Type. Returns the token type of the first alternative in the collection. However, be aware that the alternatives may represent different token types.

Declaration

protected override TokenType GetTokenType()

Returns

Type	Description
TokenType

Overrides

Token.GetTokenType()

SortByDecreasingPriority()

Sorts the set of alternatives by decreasing priority.

Declaration

public void SortByDecreasingPriority()

Implements

IEnumerable<T>

IEnumerable