tokenizing

tokenizing

美 　英

• 网络标记化；语汇单元化；标记配置

例句

This process is known more formally as parsing (or to be more precise, tokenizing, lexing, and parsing).

这个过程更正式的称呼是解析（parsing）（更准确地说，是标记解释（tokenizing）、词法解析（lexing）和语法解析）。

Method returns a scanner that simplifies the tokenizing of a string of Java code.

方法返回一个扫描程序，该扫描程序会简化对一串Java代码作标记的工作。

At the level of tokenizing, we are not yet interested in how the tokens are structured, just what they are.

在记号赋予这一层，我们还不想讨论如何构造记号，让它们维持原样就可以了。