[ragel-users] Action code for simple tokenizer?

[Seamus Abshere]

hi Adrian,

Thanks for getting back to me. I updated my action and buffering code 
for ruby:

> http://numbers.brighterplanet.com/2011/06/14/stream_parser_with_ragel_and_ruby/
> https://github.com/seamusabshere/ruby_ragel_examples
> https://github.com/seamusabshere/ruby_ragel_examples/blob/master/test/test_ragel.rb

Hopefully these examples will help people even if they don't live in the 
Ragel Guide.

Best,
Seamus

PS. For example, in Ruby, the buffering/reading loop of a scanner should 
look like:

>   CHUNK_SIZE = 1_000_000 # bytes
>   def perform
>     # So that ragel doesn't try to get it from data.length
>     pe = :ignored
>     eof = :ignored
>     %% write init;
>     # % (this fixes syntax highlighting)
>     leftover = []
>     File.open(path) do |f|
>       while chunk = f.read(CHUNK_SIZE)
>         data = leftover + chunk.unpack('c*')
>         p ||= 0
>         pe = data.length
>         %% write exec;
>         # % (this fixes syntax highlighting)
>         if ts
>           leftover = data[ts..pe]
>           p = p - ts
>           ts = 0
>         else
>           leftover = []
>           p = 0
>         end
>       end
>     end
>   end

On 6/15/11 10:28 AM, Adrian Thurston wrote:
> In the ruby code generator one uses ts and te, except they are offsets
> against 'data', instead of pointers. Aside from that, the assumptions
> and use cases are all the same.
>
> I would like to use only C in the manual. Ragel supports a number of
> languages, but it was originally designed for C and I would like the
> manual to reflect that.

_______________________________________________
ragel-users mailing list
ragel-users at complang.org
http://www.complang.org/mailman/listinfo/ragel-users