During my coding for wide finder I came across a couple of issues I want to capture:

Unix Stuff

The Bourne shell scripts I wrote work on Linux and Mac, but didn't on Solaris. But they worked find with bash, so I'm going to change the shebang to /bin/bash for all of them.

Map.def

I've been meaning to add Map.def and finally got around to it for this project:

**
** The default value to use for `get` when a key isn't mapped.
** This field defaults to null.  The value of 'def' must be immutable
** or NotImmutableErr is thrown.  Getting this field is idempotent.
** Throw ReadonlyErr if set when readonly.
**
V def

Threadlocal

I came across another use case where I really wanted a threadlocal keyword. Another one or two use cases and I'm sold Fan needs it.

Type Inference for Fields

Originally I allowed type inference to be used for fields when the right hand side was a literal. Later I changed it to force you to declare types so that the the public aspects of a class were always clearly documented. But now I rethinking that decision after writing scripts with fields such as this:

Str:Int hits := Str:Int[:] { def = 0 }

That looks like the bad old days of non-DRY Java. So now I'm thinking of allowing type inference for fields. However, I still think type inference must be with a literal versus a method call. That keeps the code easy to read without having to chain thru a bunch of calls to figure out what the type really is:

i := 0           // ok
d := 5sec        // ok
s := ""          // ok
a := Str[,]      // ok
m := Str:Int[:]  // ok
x := ["a", "b"]  // debatable
y := Sys.args    // no

Comments?

Mutable Message Passing

Fan's message passing really worked well for wide finder and performed great. I was able to easily construct different designs using sendAsync, sendSync, and join. However there were cases where I really wanted to use a mutable object as a message.

Consider the reader thread in WideFinder where I want to pass a Buf as a message. I know that once I pass the message I'm not going to reference that object anymore, but as currently designed I must pass only immutable or serializable objects as messages. What I want is the ability to use a mutable object when I know it is safe. It is kind of like a cast - just something there to remind you that you are doing something potentially unsafe.

I haven't come up with a design for this I'm satifisied with. Couple ideas:

• have a sendAsyncMutable, sendSyncMutable method; the problem is that isn't very general purpose and doesn't solve returning a mutable message from sendSync
• wrapping the mutable object in some special class such as sys::Unsafe
• maybe using some keyword with conjunction with above class
• doing something in the type system like C const correctness (but that gets ugly)

Probably my leading candidate is Unsafe, although I think it would be a bit ugly. To take a couple lines of code from wide finder:

// current where I have immutable check commented out
((Thread)msg).sendAsync(buf.seek(0))
if (msg == null) return shard

// with unsafe wrapper
((Thread)msg).sendAsync(Unsafe.make(buf.seek(0)))
if (msg == null) return Unsafe.make(shard)

Any other ideas?

OK, first, what about:

foo := SomeClass.make(args)

?

In terms of your other examples: Let me guess ...

x := ["a", "b"]  // a list of strings?
y := Sys.args    // a list of strings?

How much do types help with reading code? Can you read Python or Ruby code? Is it much harder than reading C?

I think types help more with writing code than with reading code. Types prove the absence of certain kinds of errors (the more powerfull the typesystem the more errors can be catched). Types make it easier to look things up, etc.

When you read code you assume the code is already correct. And you can see the methods called on the objects, you don't have to look up what is possible.

When you write code you normaly have tools like IDEs. They show you the type of the object or even better the avialbe methods on an object. the type checker of the compiler shows you type errors, ... .

I think I like that fields require a type. When reading the code it makes them stand out more, providing extra clarity.

class Person {
 Str surname := ""
 Int age := 0
}

I suppose it doesn't have to be the type though. It just needs some prefix that will end up being painted a different colour in editors making the fields stand out. An alternative would be a consistent prefix identifying what is a field:

class Person {
 field surname := ""
 field age := 0
}

Or a more unusual option would be to have to wrap fields in a block:

class Person {
 fields {
  surname := ""
  age := 0
 }
}

I definitely don't think that methods should automatically imply the type of a field, although I'm comfortable with it implying a local variable type.

For your case of a map initialisation, maybe it is the RHS that needs simplification?

Str:Int[] hits := { def = 0 }
// or
Str:Int[] hits := _{ def = 0 }

(where _ is some kind of implied generic type operator)

On the mutable object passing, if I've understood correctly, this is a safe operation if the caller doesn't use the object again after passing it to another thread. So, perhaps this is what should be modelled?

thread.sendAsync(buf kill)

Where kill is a keyword that kills the local variable buf from the current scope. (You could find other keyword names or operators for the concept too). Thus, a mutable object may be passed safely to another thread if it is immediately killed.

The problem with this approach is tracking if the variable has previously been stored to another variable:

temp := buf
thread.sendAsync(buf kill)
temp.doStuff()  // unsafe multithread access to buf

Anyway, its food for thought, as removing a variable from the current scope is a solution that is more general than this one use case.

Sounds like +1 for type inference on fields (with no restrictions), and a -1 too. I'm on the fence, but definitely not going to rush into any decisions. This is one of those cases where if we continue to require an explicit type, we can change our mind anytime in the future without breaking backward compatibility. But once we allow type inference we can't really go back. So I'll probably leave it as until we get more experience.

On the mutable object passing, if I've understood correctly, this is a safe operation if the caller doesn't use the object again after passing it to another thread. So, perhaps this is what should be modelled?

Stephen you got this spot on - what you'd like to do is pass a mutable object to another thread, then have some safety check that object isn't referenced again. Of course verifying that at compile time is pretty much impossible because it could have been assigned to other local variables or fields. At runtime it would require some flag and a check on every object dereference (way too much overhead). So I'm not sure how to implement something like in a general purpose way.

I agree with Stephen that fields somehow should stand out.

And about your mutable variable problem: You'd need a "linear type system" / "uniquness types" to make that safe.

what you'd like to do is pass a mutable object to another thread, then have some safety check that object isn't referenced again. Of course verifying that at compile time is pretty much impossible because it could have been assigned to other local variables or fields. At runtime it would require some flag and a check on every object dereference (way too much overhead). So I'm not sure how to implement something like in a general purpose way.

Helium mentioned "linear type system" - which I wasn't familiar with that term. But he hit the nail on the head. I saw this today on Reddit:

• http://www.malhar.net/sriram/kilim/
• http://www.reddit.com/r/programming/info/6o835/comments/

It a solution to exactly this problem. From the author's comment on Reddit:

The point is to be able to do mutable messages, yet have zero copy. The approach we have taken is to have linear ownership; only one thread gets to see a message at any time. Once it sends it, it loses visibility of the message.

That is exactly what we need for Fan message passing. I'm going to investigate when I have time. Or if anybody else has looked into it I'd appreciate feedback on how we might apply those techniques to Fan.

In a Java concurrency framework we're using (homegrown), there is a notion of view and modify permits.

In order to retrieve data from a concurrent object, you need a view permit (or maybe a modify permit), and in order to modify data in a mutable object, you need a modify permit. You can hand off a permit to another thread, explicitly, and if you do so, the permit is no longer valid for the current thread.

The permit notion is much safer and easier to understand than locks and lends itself well to message passing architectures.

That sounds interesting - I assume that requires the develop use some convention such as never storing a reference to an object within the permit?

No, since an attempt to view or modify an object without the thread having a permit to do so will throw an exception.

Actually, we've experimented with several different ways to do this, and a nice way is for all the view methods to be wrapped into the view permit, so that you get data directly from the view permit interface. Similarly, the modify methods are wrapped into the modify permit, so there's literally no way to modify the object unless you're holding a modify permit.

The full power of the approach, however, manifests itself when combined with another pervasive aspect of our concurrency framework: the raincheck. The raincheck is like a promise or future pattern, but much more sophisticated. When you ask for a view permit, you're given a raincheck on a view permit, and you can pass some code to execute when the view permit becomes available (anonymous class in Java, closure in Fan). Likewise for modify permits. So there is no blocking code.

The last detail I have been working on for some time is composite permits. You want to view N items and modify M items, all at once. You simply get a composite permit on the items, and the algorithm figures out how to do it without deadlocks.

100% deadlock-free, programmer-friendly concurrency.

We also make extensive use of immutable structures. Immutable structures are stored in watchable collections, with an additional replace operation that can be used to replace an existing element. When you first add a watcher to a collection, it's notified of everything there. So collections are never iterated through, and the listeners are guaranteed to receive notification of every addition, replacement, or removal from the collection. It works pretty well.