Using the UnderC Tokenizer Class It s often necessary to parse complex text files, where standard i/o is too clumsy. C programmers fall back on strtok(), but this can be tricky to use properly. Besides, you are still responsible for keeping strtok() fed with new input, and I don t like the schlepp. Tokenizer is a text-parsing input stream, modelled after the (undocumented) VCL TParser class, and based on the UnderC tokenizing preprocessor front-end.
