first of the 6.0 changes

[Adrian Thurston]

Hi,

I've implemented the first of the 6.0 changes. A new variable named eof and of the same type as p and pe must be declared when there are any EOF actions embedded. When there is no more input following (the last run of write exec) it should be set to pe. Otherwise it should be set to null (-1 in the case of Java and Ruby). If EOF can be known only after the input is exhausted then p=pe=eof=buf can be set and the exec code run.

The advantage of indicating EOF in this way is that EOF actions can be moved into the main processing loop and run in the same context as regular actions. They can now adjust p and jump to some other state to resume processing.

Since EOF actions can now backtrack in the input, scanners can use them to 'flush' out the last token. There is no more need to do this manually by sending trailing whitespace or some EOF character.

Adrian