[ragel-users] Re: Inline scanner

[Carlos Antunes]

Adrian,

The prefix operatior would be GREAT!

<carlos_thinking_out_loud>

And it's a incremental step toward that holy "inline scanner" grail :)

</carlos_thinking_out_loud>

Now, it would be AWESOME, if one could apply that prefix operator to
*any* machine, including scanners, obviating the need to do it
explicitly. Following your example below, one would simply do
prefix(inline_scanner), for example.

I'll be anxiously checking ragel-trunk for news!

Thanks!

Carlos

On 7/11/07, Adrian Thurston <thurs... at cs.queensu.ca> wrote:
> Hi Carlos,
>
> After much thought I've decided that inline scanners are not a suitable
> feature for ragel. It is a good technique but there are a few issues.
>
> I think that calls and returns should not be encoded in a language
> construct. They need to be treated with care and I'm not comfortable
> with encapsulating them.
>
> There is the issue of when to return. Since the feature would be using
> the scanning machinery, one might expect to be able to match multiple
> tokens as an alternative to matching just a single token. Letting the
> user implement the technique allows them to chose. Also, it allows for
> the option of erring when no pattern matches.
>
> So far all the inline machine construction operators are compositional.
> Inline scanners are not. For example you cannot safely union a machine
> that contains an inline scanner because the call necessary to invoke it
> could interfere with the other machine. This is related to the issue of
> encapsulating calls and returns.
>
> I would however be interested to add a prefix operator which would allow
> you to automatically set up the machine that the call is embedded in.
> The prefix operator would simply set every state final (this might have
> another name in the literature). After you run the prefix operator you
> could intersect it with something else to get the correct length (one in
> this case).
>
> pat = [a-z][a-z0-9]*;
>
> inline_scanner = |*
>     pat => { fret; };
>     any => { fhold; fret; };
>   *|;
>
> call_pat = ( prefix(pat) & any ) @{ fhold; fcall inline_scanner; };
>
> Regards,
>  Adrian
>
> Carlos Antunes wrote:
> > On 7/5/07, Adrian Thurston <thurs... at cs.queensu.ca> wrote:
> >>
> >>> Assuming I understand the idea you are trying to convey here (there's
> >>> a good chance that I don't), isn't it possible to transition directly
> >>> to the scanner by looking at all the potential transitions of the
> >>> scanner as if it was a "normal" state machine?
> >> Yes, that's possible. The code I sent does exactly that, only it's
> >> explicitly implemented. When it sees a whitespace character it jumps to
> >> the whitespace scanner. I'll rework what I sent so hopefully it's clear
> >> that it is an implementation of an inline scanner.
> >>
> >
> > Adrian, now that you've been able to explicitly design an inline
> > scanner, would you be able to code ragel to do it implicitly? Although
> > one can always do it by hand, as you did, it is error prone and
> > sometimes somewhat difficult, depending on what you have inside the
> > scanner. This feature would be very much appreciated.
> >
> > Thanks!
> >
> > Carlos Antunes
> >
>
>
>


-- 
"We hold [...] that all men are created equal; that they are
endowed [...] with certain inalienable rights; that among
these are life, liberty, and the pursuit of happiness"
        -- Thomas Jefferson