StringTokenizer In Java

More Utility Classes >
Siva Nookala - 06 Oct 2016

The string tokenizer class allows an application to break a string into tokens. The StringTokenizer methods do not distinguish among identifiers, numbers, and quoted strings, nor do they recognize and skip comments.

The set of delimiters (the characters that separate tokens) may be specified either at creation time or on a per-token basis.

String Tokenizer Constructors:

Constructor	Description
`StringTokenizer(String str)`	Constructs a string tokenizer for the specified string.
`StringTokenizer(String str, String delim)`	Constructs a string tokenizer for the specified string.
`StringTokenizer(String str, String delim, boolean returnDelims)`	Constructs a string tokenizer for the specified string.

An instance of StringTokenizer behaves in one of two ways, depending on whether it was created with the returnDelims flag having the value true or false:

If the flag is false, delimiter characters serve to separate tokens. A token is a maximal sequence of consecutive characters that are not delimiters.

If the flag is true, delimiter characters are themselves considered to be tokens. A token is thus either one delimiter character, or a maximal sequence of consecutive characters that are not delimiters.

A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.

A token is returned by taking a substring of the string that was used to create the StringTokenizer object.

String Tokenizer Methods:

Method	Description
`int countTokens()`	Calculates the number of times that this tokenizer's `nextToken` method can be called before it generates an exception.
`boolean hasMoreElements()`	Returns the same value as the `hasMoreTokens` method.
`boolean hasMoreTokens()`	Tests if there are more tokens available from this tokenizer's string.
`Object nextElement()`	Returns the same value as the `nextToken` method, except that its declared return value is `Object` rather than `String`.
`String nextToken()`	Returns the next token from this string tokenizer.
`String nextToken(String delim)`	Returns the next token in this string tokenizer's string.

StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead.

String Tokenizer Demo

CODE

Try it Online

import java.util.StringTokenizer;

class StringTokenizerDemo
{
    public static void main(String arg[])
    {
        StringTokenizer st1 = new StringTokenizer("Welcome to Merit" +
                        " Campus."); // LINE A
        while (st1.hasMoreTokens()) {
            System.out.println(st1.nextToken());
        }
        System.out.print("---------\n");
        StringTokenizer st2 = new StringTokenizer("It's an,Education," +
                "Website.", ","); // LINE B
        while (st2.hasMoreTokens()) {
            System.out.println(st2.nextToken());
        }
        System.out.print("---------\n");
        StringTokenizer st3 = new StringTokenizer("Learn~programming~with Java.", 
                "~", true); // LINE C
        while (st3.hasMoreTokens()) {
            System.out.println(st3.nextToken());
        }    
    }
}

OUTPUT

Welcome
to
Merit
Campus.
---------
It's an
Education
Website.
---------
Learn
~
programming
~
with Java.

DESCRIPTION

In LINE A the StringTokenizer breaks the String "Welcome to Merit Campus." into tokens and the tokens are printed. In LINE B the StringTokenizer breaks the String "It's an, Education, Website." into tokens using "," as a delimiter and the tokens are printed. In LINE C the StringTokenizer breaks the String "Learn, programming, with Java." into tokens using "," as a delimiter, includes the delimiter as returnDelims flag is true and the tokens are printed.

THINGS TO TRY

Try using hasMoreElements() and countTokens() instead of hasMoreTokens() to check for tokens.

Try using nextElement() instead of nextToken() to retrieve the next token.

Try using nextToken() along with a delimiter to retrieve the next token of the string tokenzier's string.

Try it Online

Java Collections - Utility Classes In Java

Java BitSet