tokenization

tokenization

美 　英

• 网络标记化；断词；符号化

例句

There's no need for a tokenization phase, and lookahead assertions can be used for a limited degree of context-sensitivity.

没有必要再进行符号化解析，或者用于有限度上下文敏感的前向断言。

By the way, as you will see in the examples, DParser uses no separate tokenization pass, just straight parsing.

顺便提一句，如您将在示例中所看到的，DParser不使用任何单独的标记传递，而只是直接解析。

User input may be used to adjust filtering and tokenization of the messages.

用户输入的内容可能会用于校正过滤和用于标记消息。

It supports natural language processing like word stem reduction and tokenization in multiple languages.

它支持自然语言处理，比如多种语言中的词根缩减和标记化。

Document tokenization, which is the phase that transforms a binary document into plain text, containing most of the document's content

文档标记化，它是将二进制文档转换为包含文档的大部分内容的纯文本的阶段

A Study About Chinese Dialect Identification Based on Tokenization and Language Model

基于符号化和语言模型方法的汉语方言自动辨识