#431 List Comprehensions

f00biebletch Fri 16 Jan 2009

I'd like to discuss the possibility of implementing list comprehensions within Fan. This stems from a brief discussion about the Erlang VM that made me pine for several features from Erlang.

Wikipedia has examples for the major players. Syntactically, I personally prefer the Prolog/Erlang/Haskell construct since it is closest to set builder notation as opposed to being a one line for statement in python or scala.

I don't have much of a proposal, obviously, other than "maybe make it look more like Erlang than Scala?"

JohnDG Fri 16 Jan 2009

In my view, this is part of a more general request to augment the compiler with what amounts to "syntax sugar", which suggests again the notion originally raised by Stephen of compiler plug-ins. If compiler plug-ins were supported, then we could simply write one to implement whatever feature we liked. If it turned out the feature was powerful and useful enough, then Brian and Andy might like to include it in the standard distribution.

That in mind, I'd like to propose a mechanism similar to quasiquoting in Haskell.

Overview

Each compiler plug-in exposes a very simple API.

abstract class CompilerPlugin {
   abstract Str keyword()
   abstract compiler::Node parse(Str input, 
      compiler::Expr[] arguments, 
      compiler::Node file)
}

The keyword is used by Fan to determine when the plug-in should be invoked. The parse method is used to parse an arbitrary string into a Node in the abstract syntax tree. The parse method is passed the arguments sent to the plug-in (if any), and the parsed file in which the plug-in invocation appears.

Although a compiler plug-in may parses strings into any kind of node, in some cases, only certain nodes make sense, and returning a different type of node will lead to a compile time error (generally, this is a user error and not a plug-in error).

Setup

A user installs plug-ins by moving them to the /plugins subfolder of the compiler. Plug-ins are ordinary Fan pods.

Syntax

keyword [: [arbitrary data] :]

The keyword identifies which plug-in to invoke. If there does not exist exactly one plug-in to handle the keyword, then a compile time error is generated.

Fan parses everything between [: and :] and stores it as a string. After parsing the entire file, Fan invokes any compiler plug-ins, sending them the strings and the parsed file (some way for the plug-in to query other parsed files??). The compiler plug-ins are responsible for parsing the strings into nodes in the abstract syntax tree, and returning the nodes to Fan.

Applications

  1. Software Transactional Memory
atomic [:
   // Do stuff in here, it's executed atomically
:]
  1. List Comprehensions
list := build-list [: x | x in S, isEven x :]
  1. SQL
Query query := build-sql [: SELECT * FROM Customer WHERE email='[email protected]' :]
  1. Regular Expressions
emailPattern := build-regexp [:\w+@\w+\.\w{2,4}:]
  1. Message Pattern Matching
message-match(channel, msg) [:
   msg isa EventB => doX()
   msg isa EventC && ((EventC)msg).isY() => doY()
:]

brian Fri 16 Jan 2009

John - this is a very thought provoking post. I am in the middle of working on our query engine, and considering this very problem.

Extending a Language

First let me play devil's advocate. There are a couple ways to "extend" a language:

  1. libraries only (pretty much what we have with Java and Fan today)
  2. enabling DSLs (the language itself is flexible enough to build new "languages" without introducing new syntax)
  3. compiler plugin/macros (enabling new syntax to be introduced by compiler plugins or other mechanism)

All of these approaches have their pros and cons. The library only approach is the least flexible, but results in very readable, standardized code bases. However every project requires "custom languages" often mini-formats captured in strings or config files.

The advantage of flexible syntax to enable DSLs is that one syntax defines the language. This is an advantage for tooling because tools only need to understand the one true syntax. Key ingredients for this model include: don't require parens in function calls and allowing arbitrary symbols as method names.

The compiler plugin model provides the most flexibility, but also the most rope to hang yourself with. But it can be a powerful way to embed alternate languages within the existing Fan code including the ability to capture scope like a closure does. Your use cases are all excellent examples of missing features.

General Problem

To build upon John's post, let me try to restate the problem in general terms: today you can define new libraries, but you cannot define new syntax. If we decide that is desirable, then we move in a direction where we empower people to build compiler plugins rather than extend the Fan language itself. In a way you can think of this as enabling third party APIs to have a compile time say in matters, versus being strictly regulated to runtime calls. Language similar to Fan which do stuff like this which pop up first in mind: C# does this a bit with LINQ, Boo with its syntactic macros (others????).

Syntax Thoughts

John's proposal uses a keyword and [: .... :] to capture alternate syntax. The problem with a keyword is that it can be kind of ambiguous especially with local variables and doesn't cleanly map to documentation. I'm thinking that the existing type system should be used - the keyword is replaced with a normal class (or method):

// this
Query query := build-query [: select * from customer :]

// would become
query := Query [: select * from customer :]

I like that because it seems consistent with our constructor syntax. I've also got the closure/with-block unification always in the back of my mind. So maybe the syntax looks something like:

query := Query : { select * from customer }

In which case Query is just a normal class which takes a compiler plugin as its argument just like we would pass in a closure. Or you could have arguments:

query := Query(user, pass) : { select * from customer }

Just some random thoughts, but the unification idea is still kicking around for me.

API Thoughts

Just to list out some of the possibilities. Input possibilities:

  1. string (what about comments, does core parser automatically strip them?)
  2. list of tokens

Output possibilities:

  1. string normalized Fan code which gets compiled (recursive macros?)
  2. expression tree

To keep things simple, I'm kind of inclined to say you are given a code string and return a string of normalized fan code.

jodastephen Sat 17 Jan 2009

One point we need to bear in mind is that DSLs are a hot topic right now in new languages. Scala and Groovy both claim to be good at writing DSLs.

However, I've always thought that DSLs that are based on clever tricks with the core language syntax get pretty confusing, and always end up with compromises in the allowed syntax (and can negatively impact non-DSL code). The option of a compiler plugin just strikes me of really controlling the proliferation of DSLs, and providing the full syntax control that is really necessary, with SQL being a classic example.

I really like the constructor based syntax from Brian. That looks well integrated, natural and easily controllable within Fan.

The plugin API should take an array of strings as input, as some DSLs may care about new lines. I definitely think any first version should output source Fan code (especially considering how the Javascript compiler will work).

BTW, I think fully embedded languages with no compromises could be a killer feature for Fan.

JohnDG Sat 17 Jan 2009

The compiler plugin model provides the most flexibility, but also the most rope to hang yourself with. But it can be a powerful way to embed alternate languages within the existing Fan code including the ability to capture scope like a closure does. Your use cases are all excellent examples of missing features.

Yes, and I've since thought of many others, as well -- for example, a transparent C native interface, in the style of Jeannie.

For plug-ins that work really well, you can decide whether to bundle them with the compiler. For others that prove absolutely indispensable, you could even incorporate them into the language itself, which would be essentially a matter of cut & paste (for these applications, the plug-ins can be a sort of trial grounds, where new features are invented and played with to determine their value).

I like your constructor syntax. We definitely need to support arguments to plug-ins, because some plug-ins would require compile-time Node representations of arguments, and the only way to get them is if arguments are explicitly supported (you can't pass variables to the plug-ins, only nodes, since all plug-in processing occurs at compile-time and the plug-ins are not even used at run-time).

Now for the other points, in no particular order:

  1. String input is best, unless you want to introduce a whitespace token, because some plug-ins may care about whitespace.
  2. The parser should not recognize comments or anything else until it reaches the end token. In a regexp plug-in, /*a*/ means zero or more forward slashes followed by zero or more a's followed by one forward slash. Other plug-ins may use Fan-style comments in other ways, or may introduce other kinds of comments not supported by Fan.
  3. Although I like the idea of using the feature to implement with blocks, I don't think the curly brace syntax should be used. The reason for this is that braces are very common in languages and configuration files, and the parser needs to definitively know when it reaches the end token. If you use braces for this, it means there can be no embedded braces (since some DSLs might not requiring matching open/close braces), which is a severe limitation (think of a JavaScript pre>[: :]<pre plug-in that allows you to directly tap JavaScript libraries such as jQuery from Fan, ala GWT).
  4. It's certainly possible for plug-ins to output Fan code directly. I'm not averse to this option, and it would simplify implementation of common plug-ins.
  5. Per Stephen's remark, it's not necessary to pass the plug-in a list of strings, because strings can contain embedded newline characters. The string should contain exactly the characters in between the opening symbol and the closing symbol (I've proposed pre>[:<pre and pre>:]<pre because I've never seen these symbols used outside of quasiquoting).
  6. I agree this could be a killer feature for Fan. Ruby, Scala, and Groovy-style DSLs are useful but are never quite natural looking and they cannot compete with compiler plug-ins, because compiler plug-ins give you the exact syntax you want, but with strong type checking, syntax checking, and high-performance. As one example, for a regexp plugin, you can do compile-time checking of the regular expression pattern; for SQL, you can do compile-time validation of the syntax and prepare statements in advance; etc. You basically get the best of all possible worlds.

What I like about making the compiler plug-ins ordinary classes is that you can ship them in a pod, so that the libraries you develop can rely on their own DSLs, and everything just works for the clients of the libraries.

Looking forward to seeing what comes out of this!

JohnDG Sat 17 Jan 2009

A plug-in I'd really love (and help develop):

parser := Parser [:
   <compilationUnit> :=  <using>* <typeDef>*

   <using>          :=  <usingPod> | <usingType> | <usingAs>
   <usingPod>       :=  "using" <podSpec> <eos>
   <usingType>      :=  "using" <podSpec> "::" <id> <eos>
   <usingAs>        :=  "using" <podSpec> "::" <id> "as" <id> <eos>
   <podSpec>        :=  [ffi] <id> ("." <id>)*
   <ffi>            :=  "[" <id> "]"
:]

If this could define AstUsing, etc., and all the nodes of the parser, I'd be in heaven. In fact, a plug-in like this could be used by other plug-ins for parsing.

Now to do the above, I'm not sure it's sufficient for a plug-in to return a Fan string, because Fan doesn't allow nested classes and the other things one might need to do the above. Perhaps, Brian, you can comment more.

brian Sat 17 Jan 2009

Getting down the nuts and bolts, I'm struggling with two keys issues:

  1. how the heck does an IDE do syntax highlighting
  2. how we map compiler errors back to the correct line/col numbers

Things like the ability to include a /* token in a syntax block will wreak havoc on syntax color coding of an IDE. IDEs will have to either ignore these blocks completely (which seems distasteful) or will have to have special knowledge of each specific grammar (which seems impractical). Maybe a compiler plugin doesn't do its own parsing, but rather returns a grammar which the compiler or IDEs can use. As an IDE guy yourself John what do you think?

Error handling is directly related to the output of a compiler plugin. Returning plain text source code which gets feed back thru the compiler is elegant. However error handling downstream in the pipeline will be reporting against line/col numbers which don't match up with the actual source code. That seems to strongly push us in the direction of having compiler plugins return AST trees. I really dislike that approach because the surface of the public API I must define and maintain becomes huge. Maybe the compiler plugin returns plain text but also a col/line mapping table.

I agree with you John on the notion of needing some syntax which makes for unambiguous bracketing. In order to nest these things, the compiler needs to be count opening and closing tokens, which means single tokens like [] {} () <> are out. Here is a dump of some more possibilities:

{: :}  [: :]  <: :>  (: :)
{| |}  [| |]  <| |>  (| |)
{[ ]}  [[ ]]  <[ ]>  ([ ])
{% %}  [% %]  <% %>  (% %)

I'm kind of fond of the < > combinations, probably my favorite is <|...|>.

JohnDG Sat 17 Jan 2009

how the heck does an IDE do syntax highlighting

Most of them won't, because they don't support nested grammars. However, many editors can do syntax highlighting for nested grammars; e.g. TextMate. IDEs are moving in this direction as well (at least NetBeans and IDEA; not so much Eclipse).

If someone embeds a regular expression/SQL statement/configuration datum/snippet of DSL code into a string, they don't get syntax highlighting. And if that stuff is pushed into separate files, they'll be editing it with Notepad or the like.

So the lack of syntax highlighting is not a serious drawback. I imagine that popular plug-ins would have custom support in editors/IDEs, in much the same way that IDEA has special support for Struts and GWT.

Being Fan's native editor, Flux should at least recognize the problem and support an easy way to add simple keyword/comment/literal based syntax highlighting, which is sufficient for many applications.

Now if you want to go fancy, having each plug-in return a grammar, which Fan uses for parsing, would make IDE/editor support a LOT easier. It would also give Fan intimate knowledge of the structure of the interior block, so that it no longer becomes an issue what symbol we use for opening/closing (even { and } would work). Moreover, it pushes the likely duplication of parsing/error reporting from the plug-ins into Fan, which would result in a much lower barrier of entry to creating a plug-in.

What makes this option viable is Fan's support for dynamic programming, so you can construct a dynamic syntax tree, where the nodes have names that correspond to productions in the grammar (root->compilationUnits).

The only thing I don't like about this approach is that it will take much more time to implement. You need a nice generalized parser, for example, a packrat parser for PEGs (very powerful with a simple implementation; add support for left-recursion to make grammar writing trivial, albeit with added complexity). Now since you're in the middle of writing a query engine, maybe you can subsume a generalized parser in your work.

However, it's clearly the Cadillac solution that will result in the largest number of plug-ins and the easiest integration with IDE/editors.

how we map compiler errors back to the correct line/col numbers

For now, let's assume the plug-in will do parsing/error reporting since that makes the Fan implementation much easier.

The generated Fan code won't have any compiler errors if the plug-in is working correctly (for debugging a plug-in, however, it would be helpful if the compiler showed the offending generated code). There are only two kinds of errors:

  1. Errors in the DSL. In this case,the plug-in will throw the appropriate exception that indicates the line/col number of the offending token.
  2. Errors in the placement of the invocation. For example, if a plug-in does not return an Expression, then it's absurd to assign a variable to the plug-in block (e.g. myVar := Foo <| ... |>). The Fan compiler can easily generate messages that only reference the types of the expressions and their locations.

    I really dislike that approach because the surface of the public API I must define and maintain becomes huge.

Now it seems to me, you're already going to have to be maintain a large public API, at least sufficient to cover the entirety of Fan's AST, because the main benefit of putting DSLs in Fan is that they can have full access to the context in which they appear -- access variables and functions, etc. This is not an argument to have plug-ins return AST, because in retrospect, I think I've become convinced it's easier to write a plug-in that returns Fan code than trying to wire together a valid AST. Just an FYI.

freddy33 Sat 17 Jan 2009

For Info: IntelliJ support nested grammar, and FFI perfectly. You can declare a literal type in one language to be "Chameleon", which will call another language parser, but still have the variables scope shared. This how they do SQL strings in Java, and groovy string ${x.foo()} code completion. Sorry, did took the time to read the whole discussion, from a high point of view I'll say: Flexible but limited (in scope and feature) integration of language plugins is always good (make your compiler cleaner, and provide some way to test something before making it mainstream). But, in the case of Fan I really feel that having many clean pods API wrapping cool libraries of Java or .NET platform is more important, and will give you great idea about what features are the more critical.

My 2cts on this.

PS: I like the <| |> for writing regex. PPS: Now should Regex be a plugin?

brian Sat 17 Jan 2009

Yeah I think <|...|> works best for me too.

John, it is unclear to me that phase of the pipeline you are thinking about. I was thinking of along the lines that plugins are run at the parsing phase before we've resolved everything into types, etc. Fan is a little unusual in that I know the difference b/w type identifiers and other identifiers at parse time, this is how I deal with some of the grammar's ambiguity. Not sure how that gets fed to the plugins.

Couple design options:

  1. Plugins are run at parse time, merely producing plain text that gets feed back into the parser. The output of this phase then gets fed thru the rest of the existing pipeline for error checking, etc. In this design the only error checking the plugin does it syntactic checks. All type checking happens in the Fan compiler (most error checking happens in CheckErrors).
  2. Plugins are parsed after the rest of the pipeline has completed. This requires that we know the type of the pluggable syntax expression. The plugin then could access the AST with everything correctly typed. In this case, all error reporting would be the responsibility of the plugin.
  3. Plugins get integrated into the full compiler pipeline getting hooks at all the various phases such as parsing and error checking.

I was definitely thinking about option 1. where the plugin is really just a syntactic translation to Fan code.

JohnDG Sat 17 Jan 2009

I have always assumed option (2), possibly with some slight modification: the compiler doesn't do type checking for each "hole" until after invoking the plug-in (though it seems simpler with no loss of generality to just require the plug-in to export the type of node it returns).

Now I have no problem with letting the compiler do all type checking on the hunk of returned code (which does necessitate a translation table of some sort), but if the types are not available to the plug-in, it limits the kinds of plug-ins you can write. You can do a basic SQL plug-in that simply invokes toStr() on anything it sees (hoping for the best), but you can't do one that provides basic ORM (for example).

In general, I think not having the type information will lead to poorer error reporting and will severely limit the kinds of plug-ins that can be written.

jodastephen Sat 17 Jan 2009

I suspect that option 1 (parse time) will be sufficient provided that the full list of type identifiers and variable types is available. Brian, you seem to be indicating that is the case? If so, then I'd expect the plugin to always be able to output valid fan code, and thus to take responsibility for returning parse errors in its input.

I can't say I like <| |>, or any of the double character symbols. Would it not be possible for the plugin to determine when the end of the plugin specific code is (by being given the whole of the rest of the file as input)?

brian Sat 17 Jan 2009

I suspect that option 1 (parse time) will be sufficient provided that the full list of type identifiers and variable types is available. Brian, you seem to be indicating that is the case?

I need to think about it. If you consider any type of complex DSL which allows arbitrary Fan expressions to be nested (which might capture from the lexical scope) then you are going to need to implement (or reuse) Fan's expression grammar. Or alternatively use something like ${expr} to pass a expression back to the Fan compiler. But I don't see in these cases why a compiler plugin needs anymore information than the Fan compiler needs for parsing. Parsing is based strictly on the lexical structure.

I think the issue is really what happens further down the pipeline. Lets say a compiler plugin translates into standard Fan code. Then the Fan error checking should capture many type errors. However those errors may be confusing and may not handle DSL specific errors. So ideally even if the plugin operates during the parser phase, you'd probably like to give it a hook for error checking after symbols have been resolved. You might even let plugins generate their own AST nodes and assembly process (but that seems a bit too open ended).

In some ways this isn't too different from what FFI plugins do - see CBridge. It actually works with the AST directly and gets hooks during type checking to report errors or perform implicit coercions.

I can't say I like <| |>, or any of the double character symbols. Would it not be possible for the plugin to determine when the end of the plugin specific code is (by being given the whole of the rest of the file as input)?

I think it is critical that tools can skip extended code blocks without having knowledge of the specific plugins. Otherwise it would be hopeless for IDEs to do basic color coding. So I think a standardized open/close symbol makes sense.

JohnDG Sat 17 Jan 2009

But I don't see in these cases why a compiler plugin needs anymore information than the Fan compiler needs for parsing.

It needs access to the AST, or else faces the job of itself parsing the Fan code (it cannot simply call the Fan compiler because doing so will result in an infinite loop). Consider an STM plug-in: the variables/methods that appear in an Atomic block are all grabbed from the context:

Atomic <|
   listeners.add(newListener) // listener is field of class, newListener is param

   ...
|>

The code generated by the plug-in depends intimately on the types of the variables involved.

Similarly, suppose one plug-in allows you to filter SQL results using ordinary closures -- not by grabbing everything from the database and running the filter, but by understanding what the filter is doing and translating it into SQL. For example:

myFunc := |Customer c -> Bool| { return c.firstName == 'joe'; }

DBQuery <| 
   select * from Customers
   filterBy myFunc
|>

The above actually generates an efficient SQL query.

More ambitiously, how about LINQ for Fan?

maxAge := 20

LINQ <|
   results :=  from c in Customers
               where c.age < ${maxAge}
               select new {c.id, c.fullName};
|>

All of the above examples and many more require that plug-ins have access to the AST of the surrounding context (and possibly, of all code!). They need full type information. Letting them have a list of variables, methods, and fields isn't sufficient.

I think it is critical that tools can skip extended code blocks without having knowledge of the specific plugins. Otherwise it would be hopeless for IDEs to do basic color coding. So I think a standardized open/close symbol makes sense.

Exactly. It's also necessary for approach #2 (the one required for the above kinds of plug-ins, and ones like them), because you need to essentially do all parsing and resolution before invoking the compiler plug-ins in a 2nd pass.

JohnDG Sat 17 Jan 2009

Lets say a compiler plugin translates into standard Fan code.

If the AST is available to all plug-ins, then they have the option of pushing any type errors to the later stage (when the string output of the plug-in is compiled as Fan code), or doing the type analysis themselves (useful for strong plug-ins designed to be used a lot and written and maintained by a community of developers). So you get the ability to write fast and dirty plug-ins but also do the really nice models, such as what you'd want for a LINQ-style plug-in.

helium Sun 18 Jan 2009

A thing that might happen:

The python programmer whats his comprehensions and creates

result := for <| x * x for x in xs if x > 0 |>

the Erlang programmer likes his version

result := comprehension <| x * x || x <- xs, x > 0 |>

the C# programmer is used to his LINQ syntax

result := linq <| from x in xs where x > 0 select x * x |>

and the Haskell programmer wants a more general syntactic suggar for monadic code

result :=  do <| x <- xs; guard x > 0; return x * x |>

.

...

f00biebletch Sun 18 Jan 2009

In helium's example, how would you type result? Would it be like an anonymous type in .NET? How does Haskell's type system handle this sort of thing?

brian Sun 18 Jan 2009

A thing that might happen:

As soon as a language provides good support for DSLs, it seems inevitable that the language will splinter in dialects. I can see pros and cons.

For example should we be working on a solid list comprehension syntax that everything uses in the core compiler, or figuring out how to enable everyone to do their own special syntax?

jodastephen Sun 18 Jan 2009

For example should we be working on a solid list comprehension syntax that everything uses in the core compiler, or figuring out how to enable everyone to do their own special syntax?

While I suspect that Fan will eventually have core list comprehensions, I don't find it an immediate priority. Allowing others to drive their own DSLs is a way of democratising Fan and encouraging participation - as well as a way to allow for some really cool features (SQL, XML, JSON, concurrency, ...)

JohnDG Sun 18 Jan 2009

A thing that might happen:

Helium, I don't think that will happen, at least not to the extent and not in the way that you imagine. We'll see a few variations of list comprehensions and over time they'll compete with each other, which will benefit the developers using them.

In helium's example, how would you type result?

It would be a list, since this is a list comprehension. More generally, the type that you get out from the plug-in depends on the types that you feed it.

As soon as a language provides good support for DSLs, it seems inevitable that the language will splinter in dialects. I can see pros and cons.

I don't think it will happen anymore than it already does with libraries. For any given area, there are several libraries that developers can choose from. I don't think that's a bad thing. Experimentation is the community's way of evolving the best possible design -- and it yields results far superior to up front design, and more tailored to the needs of each subcommunity.

For example should we be working on a solid list comprehension syntax that everything uses in the core compiler, or figuring out how to enable everyone to do their own special syntax?

The latter is far more valuable. If you design a list comprehension syntax, you won't know how good it is until you start seeing lots of code written using it. Chances are, you'll make mistakes, and force people down a path that may be far from optimal. Plus, you'll be spending your time doing something the community could be doing.

You can make your above argument for everything: regular expressions, JSON, SQL, XML, STM, embedded Javascript and other scripting languages, property testers, etc. The list goes on. Should all of them be in core???

I'm in favor of a more democratic process. Give the language the ability to extend itself, in the relatively straightforward fashion we've discussed (which doesn't involve, for example, allowing the user to arbitrarily extend core Fan syntax; it isolates changes to well defined areas). Then let the community develop the plug-ins they need, and evolve them based on feedback from actual use. If something proves extraordinarily useful, then maybe it's time to push it into the core. And once it's developed and field-tested, and already well integrated into the compiler pipeline via the plugin architecture, porting it into Fan will be a piece of cake.

brian Mon 19 Jan 2009

OK, I can't say I'm 100% sold that we should definitely do this, but I am in strong enough agreement that I should spent effort on a prototype.

Since I am the process of designing an embedded query language for Folio (our distributed database for Bespin), I will use that as my use case along with Regex and see where this goes.

I do think list comprehensions are important though, so I want to tackle them at some point. In fact I've always envisioned something like LINQ for Fan which could potentially provide access to the AST for SQL or XPATH generation. But the compiler plugin feature we are talking about here might subsume all that.

JohnDG Mon 19 Jan 2009

Since I am the process of designing an embedded query language for Folio (our distributed database for Bespin), I will use that as my use case along with Regex and see where this goes.

Sounds good. I can't wait until it's in beta. My first plug-in will likely be a shell plug-in since I've often wanted one when scripting. e.g.:

out := Shell <|
   #!/bin/sh

   ls ${dir}
|>

And it's simple enough to knock out in a day while still hitting all major operating systems.

But the compiler plugin feature we are talking about here might subsume all that.

Exactly. You can do LINQ and lots of other things strictly through the plug-in architecture, and in a very seamless way (the plug-in invocation syntax is non-invasive and lightweight). That is, assuming the plug-ins do have access to the AST, which I'm hoping is the option you take.

brian Tue 20 Jan 2009

OK, now comes the part of software development which always sucks - naming!

What should we call this feature? A couple aspects of how we will use the name:

  1. the overall feature
  2. key class names like FooPlugin
  3. the pluggable code embedded in Fan between the <| |>

My vote is that this feature is called DSLs (Domain Specific Languages). So in talking about this example:

re := Shell <|ls ${dir}|>

We would have something like ShellDslPlugin, and would call the string ls ${dir} embedded into Fan a DSL.

Does everybody like that name? Any other suggestions?

JohnDG Tue 20 Jan 2009

Calling the feature DSLs will be great for publicity. :-)

I agree with your naming conventions. Technically, the string should probably be called a Domain Specific Language Snippet (DSLS), but that may be overkill and the extra precision is not really required.

One thing I'm curious on: how are you going to associate the plug-in ShellDslPlugin with the Shell? Is Shell a method on ShellDslPlugin?

In the end, I'd just like the flexibility for lower-case invocators; e.g.: shell instead of Shell.

Login or Signup to reply.