Enum TokenizerFlags

Flags controlling tokenizer behaviour

Namespace: SdlSdl.LanguagePlatformCoreTokenization

Assembly: Sdl.LanguagePlatform.Core.dll

Syntax

[Flags]
[DataContract]
public enum TokenizerFlags

Fields

Name	Description
AllFlags	All flags set
BreakOnApostrophe	If true, break tokens on apostrophe characters
BreakOnDash	If true, break tokens on dash characters
BreakOnHyphen	If true, break tokens on hyphen characters
BreakOnSymbol	If true, breaks tokens for a defined list of symbols
DefaultFlags	Default flag values May differ from AllFlags if further flags are added
NoFlags	No tokenizer flags set