com.voytechs.jnetstream.npl
Class ExpTokenizer

java.lang.Object
  com.voytechs.jnetstream.npl.TokenList
      com.voytechs.jnetstream.npl.ExpTokenizer

public class ExpTokenizer
extends TokenList

Field Summary

Fields inherited from class com.voytechs.jnetstream.npl.TokenList

in

Constructor Summary

ExpTokenizer()

ExpTokenizer(java.io.Reader reader)

ExpTokenizer(java.io.Reader reader, java.lang.String filename)

Method Summary

Token	fetchRawTokenFromStream() We override this method so we can allocate the Expanded ExpTokens.
protected Token	fetchTokenFromStream() Fetches a token from stream and translates the stream token(s) into cutom token.
java.lang.String	getFilename() Returns the filename of the stream being read.
static void	main(java.lang.String[] args) Test function for ExpTokenizer
Token	nextKeywordToken()
Token	nextKeywordToken(java.lang.String expectedKeyword)
Token	nextWordToken()
Token	peekWordToken()
Token	translateNextToken() Do all possible translations with tokens in the buffer or the stream.
protected java.lang.String	translateToken(java.lang.Object[] keys) Translate tokens on the stream into custom tokens with extend types.

Methods inherited from class com.voytechs.jnetstream.npl.TokenList

consumeTokens, getLineNumber, hasMoreTokens, insert, lookAhead, lookAhead, lookAheadNoTranslation, nextToken, nextTokenNoTranslation, previousToken, pushBack

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExpTokenizer

public ExpTokenizer()

Parameters:

-

Throws:

ExpTokenizer

public ExpTokenizer(java.io.Reader reader)

Parameters:

-

Throws:

ExpTokenizer

public ExpTokenizer(java.io.Reader reader,
                    java.lang.String filename)

Parameters:

-

Throws:

Method Detail

nextKeywordToken

public Token nextKeywordToken(java.lang.String expectedKeyword)
                       throws IdentifierSyntaxError

Throws:

IdentifierSyntaxError

nextKeywordToken

public Token nextKeywordToken()
                       throws IdentifierSyntaxError

Throws:

IdentifierSyntaxError

nextWordToken

public Token nextWordToken()
                    throws IdentifierSyntaxError

Throws:

IdentifierSyntaxError

peekWordToken

public Token peekWordToken()
                    throws IdentifierSyntaxError

Throws:

IdentifierSyntaxError

translateToken

protected java.lang.String translateToken(java.lang.Object[] keys)

Translate tokens on the stream into custom tokens with extend types. Stream tokens may need to be combined for custom tokens. i.e. "=" combined with another "=" token to create a "==" token which is used in cutom mapping. The rule is that tokens off the stream will be combined until there is no longer a match. The longest match possible returned.

translateNextToken

public Token translateNextToken()

Do all possible translations with tokens in the buffer or the stream.

Specified by:

translateNextToken in class TokenList

fetchTokenFromStream

protected Token fetchTokenFromStream()

Fetches a token from stream and translates the stream token(s) into cutom token.

Overrides:

fetchTokenFromStream in class TokenList

Returns:

Returns the translated token.

fetchRawTokenFromStream

public Token fetchRawTokenFromStream()

We override this method so we can allocate the Expanded ExpTokens. This means that everything comming from this tokenizer's nextToken() method can be cast to ExpToken.

Overrides:

fetchRawTokenFromStream in class TokenList

getFilename

public java.lang.String getFilename()

Returns the filename of the stream being read.

Specified by:

getFilename in class TokenList

main

public static void main(java.lang.String[] args)

Test function for ExpTokenizer

Parameters:

args - command line arguments