Warm tip: This article is reproduced from stackoverflow.com, please click
java parsing token

Are ">>"s in type parameters tokenized using a special rule?

发布于 2020-04-20 11:18:20

I'm confused by the Java spec about how this code should be tokenized:

ArrayList<ArrayList<Integer>> i;

The spec says:

The longest possible translation is used at each step, even if the result does not ultimately make a correct program while another lexical translation would.

As I understand it, applying the "longest match" rule would result in the tokens:

  • ArrayList
  • <
  • ArrayList
  • <
  • Integer
  • >>
  • i
  • ;

which would not parse. But of course this code is parsed just fine.

What is the correct specification for this case?

Does it mean that a correct lexer must be context-free? It doesn't seem possible with a regular lexer.

Questioner
Matt Fenwick
Viewed
48
分享
Matt Fenwick 2013-05-29 10:03

Based on reading the code linked by @sm4, it looks like the strategy is:

  • tokenize the input normally. So A<B<C>> i; would be tokenized as A, <, B, <, C, >>, i, ; -- 8 tokens, not 9.

  • during hierarchical parsing, when working on parsing generics and a > is needed, if the next token starts with > -- >>, >>>, >=, >>=, or >>>= -- just knock the > off and push a shortened token back onto the token stream. Example: when the parser gets to >>, i, ; while working on the typeArguments rule, it successfully parses typeArguments, and the remaining token stream is now the slightly different >, i, ;, since the first > of >> was pulled off to match typeArguments.

So although tokenization does happen normally, some re-tokenization occurs in the hierarchical parsing phase, if necessary.

johnchen902 2013-05-31 07:34:59

Why compiler don't re-tokenize things such as a--b?

Matt Fenwick 2013-05-31 11:12:39

@johnchen902 why should it? Retokenization isn't a general strategy for rescuing failed parses, it's only used in one special case so that you don't have to write A<B<C> >.

Related issues

1
Alternative approach for checking data types to build an object
2
Is it possible to get first and last name from Google Authentication on Firebase?
3
Constructor that receives TXT files to read and store them
4
Logback with Elastic Beanstalk
5
Camel Predicate Example in xml DSL
6
How do I get user input validation into my java calculator program?
7
firebase getting exact childs from database
8
Listener for custom dialog null
9
JFrame very small
10
AWS Elastic Beanstalk Application Logging with Logback