Parsing

In order to implement the algorithm from the previous section, we need to be able to traverse a string and break it into operands and operators. This process is an example of parsing, and the results---the individual chunks of the string---are called tokens.

Java provides a built-in class called a StringTokenizer that parses strings and breaks them into tokens. To use it, you have to import it from java.util.

In its simplest form, the StringTokenizer uses spaces to mark the boundaries between tokens. A character that marks a boundary is called a delimiter.

We can create a StringTokenizer in the usual way, passing as an argument the string we want to parse.

The following loop is a standard idiom for extracting the tokens from a StringTokenizer.

The output is

For parsing expressions, we have the option of specifying additional characters that will be used as delimiters:

The second argument is a String that contains all the characters that will be used as delimiters. Now the output is:

This succeeds at extracting all the operands but we have lost the operators. Fortunately, there is one more option for StringTokenizers.

The third argument says, "Yes, we would like to treat the delimiters as tokens." Now the output is

This is just the stream of tokens we would like for evaluating this expression.