lexer

lexer

美 　英

• 网络词法分析器；词法分析程序；词法分扫描器类

例句

The lexer actually does some of the work of figuring out where in a message it is, but the parser still ties everything together.

lexer实际上完成了指出它位于消息中的哪个地方的一些任务，但是解析器仍将所有内容放在一起。

Additionally there is implemented a IDL lexer, which allows to use the preprocessor library as the lexing component of a IDL oriented tool.

此外还实现了一个IDLlexer，允许将预处理器库用作某个面向IDL的工具的lexing组件。

Whenever possible, try to develop separate tests for your lexer and your parser.

只要有可能，请尝试为您的lexer和解析器开发单独的测试。

The lexer for this example is more complicated than the one for the calculator; quite a bit so.

该示例中的lexer比计算器的更为复杂；复杂得多。

So this lexer does not recognize any keywords (except true and false, which are needed by the preprocessor itself).

因此，这个lexer不识别任何关键字（除了true和false，因为预处理器本身需要它们）。

A lexer is a software component that divides text strings into individual words, or tokens, so that the individual words can be indexed.

lexer是一个能将文本字符串分成单个命令，或者标记的软件组分，这样使得单个命令可以被检索到。

On the basis of the test tool of executing characteristics of programs, we modify the lexer and parser to implement it.

该工具是在程序执行特征测试工具的基础上，将其词法分析器和语法分析器加以修改而实现的。

These directives are used to define the tokens the lexer can return.

这些指令用来定义lexer可以返回的记号。

The lexer is the part of our language knowledge that says "this is a sentence; this is punctuation; twenty-three is a single word. "

词法分析器是我们的语言知识中识别“这是一个句子；这是标点；twenty-three是一个单一的词”的那一部分。

The symbols produced by the lexer are called terminals or tokens.

lexer生成的符号称为终结符（terminals）或者记号（tokens）。

JAPANESE_LEXER: A lexer for extracting tokens from Japanese text.

JAPANESE_LEXER：一个从Japanese文本在中提取标记的lexer。

The C++ lexers packaged with the Wave library may be used standalone, too, and are not tied to the C++ preprocessor iterator at all.

与Wave库一起打包的C++lexer也可以单独使用，并没有与C++预处理迭代器捆绑在一起。

So, in this example, NUMBER is a terminal; the lexer is producing this result.

所以，在这个例子中，NUMBER是一个终结符；lexer正是生成这种结果。

KOREAN_MORPH_LEXER: A lexer for extracting tokens from Korean text.

KOREAN_MORPH_LEXER：一个从Korean文本在中提取标记的lexer。

The tokens ALPHA, INT, and REAL are typically provided from the lexer.

INT和REAL通常由lexer提供。

KOREAN_LEXER: A lexer for extracting tokens from Korean text.

KOREAN_LEXER：一个从Korean文本在中提取标记的lexer。

This does mean that the lexer and the parser both have to know that an empty line separates a header from a body.

这确实意味着lexer和解析器都必须明白，是一个空行将标题从主体中分离了出来。

A listing of supported lexer types follows.

下面是一个支持lexer类型的清单。

JAPANESE_VGRAM: A lexer for extracting tokens from Japanese text.

JAPANESE_VGRAM：一个从Japanese文本在中提取标记的lexer。

Most simple programming projects, of course, can get by with very trivial lexers.

当然，大部分简单编程项目使用非常普通的lexer就可以完成。

CHINESE_VGRAM: A lexer for extracting tokens from Chinese text.

CHINESE_VGRAM：一个从Chinese文本在中提取标记的lexer。

Thus, P: : RD on top of Perl 5 is a powerful parser and lexer combination.

因此，Perl5之上的P：：RD是一个解析器与词法分析器的强大组合。

A good lexer example can help a lot with learning how to write a tokenizer.

一个好的lexer例子会非常有助于学习如何编写断词器（tokenizer）。