#1670 Java byteCode on Android

go4 Sun 9 Oct 2011

I can't run the fantom on Android by JarDist. I compare the java byte code again and again. Finally, I find the fantom not set the ACC_SUPER flag.

My code:

class Widget
{
  virtual Void paint()
  {
    echo("widget")
  }
}

class Button : Widget
{
  override Void paint()
  {
    echo("button")
    super.paint()
  }
}

//in java
Button b = Button.make();
b.paint();

Error:

button //print button
// this exeception occur on invokespecial instructions: super.paint();
java.lang.NoSuchMethodError: Widget.paint

brian Sun 9 Oct 2011

Promoted to ticket #1670 and assigned to brian

Okay, I need to investigate that. Have you tried setting that flag in Java fanx.emit code to see if it resolves the problem?

go4 Sun 9 Oct 2011

I don't know how to get it fixed. I will verify that if you give a patch.

Thanks.

go4 Sun 9 Oct 2011

I find another issue:

invokespecial instruction can only be used to invoke a private method and super method. But the fantom not really have private method. so there is expected invokevirtual instruction.

I don't know why Android is so careful on java byte code. But it will be sweet if fantom could run on Android.

brian Mon 10 Oct 2011

I don't think that error has anything to do ACC_SUPER actually now that I've read about it. But if you want to try it, you can set that flag in FTypeEmit.java line 73, something like this:

init(jname(type.self), base(), mixins(), jflags(type.flags)|SUPER);

If you want to try that we can see if it works or not, then take the problem from there.

go4 Mon 10 Oct 2011

It bring more problems.

In Java Virtual Machine implementations prior to version JDK 1.02, this instruction was called invokenonvirtual, and was less restrictive than invokespecial - it wasn't limited to invoking only superclass, private or <init> methods. The class access flag ACC_SUPER (see Chapter 4) is used to indicate which semantics are used by a class. In older class files, the ACC_SUPER flag is unset. In all new classes, the ACC_SUPER flag should be set, indicating that the restrictions enforced by invokespecial are obeyed. (In practice, all the common uses of invokenonvirtual continue to be supported by invokespecial, so this change should have little impact on JVM users).

This means that set ACC_SUPER will change the semantics of invokespecial.

brian Mon 10 Oct 2011

Your code example is the typical example of where you have to use invokespecial - it is the only way to invoke Widget.paint non-virtually. I don't think ACC_SUPER does anything but make use of invokespecial more restrictive. And I think things already pretty strict because that is why I can't allow super calls to mixins.

What I would suggest is dump the Java bytecode from DistJar for those two classes and see if you notice anything different b/w Java. It should be exactly the same bytecode so not sure why it doesn't work.

go4 Mon 10 Oct 2011

The Android don't complain NoSuchMethodError when I set the ACC_SUPER flag. But other issues raised( the fwt can't work correctly ).

The invokespecial is different from invokeNovirtul. invokespecial is used to invoke:

  • the instance initialization method, <init>
  • a private method of this
  • a method in a superclass of this

invokespecial reference

All Java compilers since JDK 1.1 always set the ACC_SUPER flag.

Why ACC_SUPER

And Android only support jvm bytecode 1.4-1.6.

brian Mon 10 Oct 2011

I don't believe we do anything that prevents us from setting the ACC_SUPER flag - we don't use invokespecial except in the three cases you listed (and limit Fantom at the language level as a work around where we truly need an invokenonvirtual opcode).

So when you say fwt doesn't work correctly what does that mean? FWT is all SWT based, how would it ever work in Andriod?

brian Mon 10 Oct 2011

I ran thru the entire test suite using the ACC_SUPER flag, and there is only one thing that really makes a difference: using a named super to call a method which isn't your direct parent:

class A   { virtual Str x()  { "A.x" } }  
class B : { override Str x() { "B.x" } }
class C : { override Str x() { "C.x" } }

In Fantom we allow C to call A.super.x to explicitly use class A's version. The ACC_SUPER flag prevents that and forces you to only be able to call B.x.

It does appear that we use that feature in fwt::ContentPane to override add in one place.

go4 Tue 11 Oct 2011

invokespecial can be used to call a private method of this.

But, the fantom's private method mapping to java's package method. In fact, have no private protection in the emitted java bytecode. See the fanx/emit/FTypeEmit.java, line 373.

In this case, the Android refuse to call noPrivate method by invokespecial.

go4 Sat 15 Oct 2011

I add a CompilerStep to rename the private slot. The private slot foo is renamed to foo$podName$typeName. It works well on Android, although there is little issue on reflection and named super.

Code:

class RenamePrivate  : CompilerStep
{

  new make(Compiler compiler) : super(compiler) {}

  override Void run()
  {
    log.debug("RenamePrivate")
    walk(compiler, VisitDepth.expr)
  }

  override Void visitMethodDef(MethodDef def)
  {
    if (needRename(def)) def.name = newName(def)
  }

  override Void visitFieldDef(FieldDef def)
  {
    if (needRename(def)) def.name = newName(def)
  }

  private Bool needRename(CSlot def)
  {
    if (def.isPrivate && !def.isStatic && !def.isCtor && !def.isNative &&
        !def.name.startsWith("instance\$init"))
    {
      return true
    }
    return false
  }

  private Str newName(CSlot def)
  {
    "$def.name\$$pod.name\$$def.parent.name"
  }

  override Expr visitExpr(Expr expr)
  {
    if (expr is CallExpr)
    {
      c := expr as CallExpr
      if (needRename(c.method)) c.name = newName(c.method)
    }
    return expr
  }

}

compiler/fan/assembler/CodeAsm.fan, line 1287:

-if (targetId == ExprId.superExpr ||
-(targetId == ExprId.thisExpr && !m.isVirtual && !m.parent.isObj))
+if (targetId == ExprId.superExpr)

go4 Sun 16 Oct 2011

I mean, if using invokevirtual instead of invokespecial to call a private method, private slot name conflicts will occur. So, I try to do name mangling for private slot.

You know, all things is public in javascrpt. There is also have namespace problem. See the `http://fantom.org/sidewalk/topic/1651#c11356`

brian Tue 18 Oct 2011

There are several issues:

  1. it appears that Andriod requires ACC_SUPER which means we must remove support for named supers above your direct base class
  2. it appears that Andriod won't let you use invokespecial on a package scoped method, so we will need to switch to invokevirtual.

It appears that Java correctly dispatches invokevirtual with package private methods. But I couldn't find it in the JVM spec, so I've asked the experts.

Just to list some of the critical issues:

  • private methods are not emitted as private Java methods because then closures can't access them without method wrappers (what Java does for inner classes)
  • both internal and private methods have potential for inter-pod naming conflicts which hopefully can be solved with JVM package private dispatch behavior
  • JS will likely require name mangling, but I'd like to avoid that at compiler level and keep it a target runtime issue

go4 Tue 18 Oct 2011

Very exciting summary. Thanks.

Actually, Dalvik(Android VM) is not a JVM. The compiler translate the jar to Dalvik .dex file:

fcode => java bytecode => dex file.

The Dalvik have two opcode for invoke novirtul: invoke-supper and invoke-direct. It is a pity, they have the same limits as invokespecial.

DanielFath Tue 18 Oct 2011

So we need another target, no?

fcode => dex files.

brian Sat 22 Oct 2011

Just to note: John Rose did confirm the way invokevirtual works with package scoped methods. This means we can continue to emit private/internal methods to package scoped methods in the JVM without resorting to name mangling.

So I think the only thing we lose is the ability to use a named super more than one level above your direct base class. Other than that, we should be able to safely switch to ACC_SUPER and invokevirtual on private method methods.

brian Mon 14 Nov 2011

@go4

I have pushed the first two changes required for Andriod:

I have not made the final required change which is to switch non-virtual method calls from invokespecial to invokevirtual. This can be tested in a fairly safe way by changing compiler::CodeAsm.fan around line 1286 like this:

targetId := call.target.id
if (targetId == ExprId.superExpr)
  op(FOp.CallNonVirtual, index)
else
  op(FOp.CallVirtual, index)

Can you test that out in conjunction with the other changes and see if that gets us where we need to be for Andriod?

That change is mostly safe, but it creates some loop holes in that private methods inside the same pod might not resolve correctly. So if we really go down this path we'll have to tackle the whole clusterfuck of generating public wrapper methods around private methods for closure access so we can actually call private methods nonvirtually. But I don't want to go down that path until you have verified that this will all work on Andriod and we are sure this is the right path.

go4 Wed 16 Nov 2011

Yes, It works.

In fact, I have successfully used Fantom on Android more than a month. It works very well, Thanks.

There are two test failed:

testSys::CallTest.testNonVirtual
testSys::CompilerJsTest.testMethodHiding

I don't think this is a big problem. just a compiler warning is enough.

Perhaps all the problems are a Android VM bug.

brian Wed 16 Nov 2011

Those tests are the ones that actually check that private methods aren't being treated as virtual :-) So that would be expected with the change from CallNonVirtual to CallVirtual.

So other than that, the full test suite passes on Andriod? That is pretty sweet.

I'm going to post a build this week, but probably won't officially make this change until following build. The remaining issue is now this:

  1. We can't make private methods actually be private because then synthetic closure classes can't access the private methods (Java generates wrapper methods for inner classes to handle this); for Fantom I emit private methods as package private
  2. Andriod doesn't let us use invokespecial on package private methods
  3. If we switch to use invokevirtual, then it is possible that a subclass which creates a private method with the same name as a superclass in the same pod will be incorrectly treated an override
  4. Potentially this issue is related to JavaScript method hiding for internal and private methods

There are three basic strategies I see we can use:

  1. Emit Fantom private methods as private Java methods and generate internal wrappers for closure use (like Java does for inner classes)
  2. Disallow subclasses to reuse internal or private method names which would override a method in a parent class in the same pod (treat these as compiler errors and prevent it happening in first place)
  3. Name mangle all private and internal methods (this might have benefit of helping JavaScript issues)

I'm sort of debating between B and C myself. But comments welcome

go4 Thu 17 Nov 2011

I just run the testSys.

I can't run the test by JarDistEnv:

java -jar testSys.jar testSys
sys::Err: Cannot call '[java]fanx.tools::Fant.fanMain': java.lang.ArrayIndexOutO
fBoundsException: 0
  fan.sys.Method.invoke (Method.java:588)
  fan.sys.Method$MethodFunc.call (Method.java:252)
  fan.sys.Method.call (Method.java:141)
  fanjardist.Main.main (Main.java:17)

The some IO failed on Android:

Failed:
  testSys::BufTest.testEquality
  testSys::BufTest.testBasicIO
  testSys::BufTest.testSizeCapacity
  testSys::BufTest.testSlice
  testSys::BufTest.testDup
  testSys::BufTest.testWriteBuf
  testSys::BufTest.testReadBuf
  testSys::BufTest.testBinary
  testSys::BufTest.testConveniences
  testSys::BufTest.testChar
  testSys::BufTest.testUnread
  testSys::BufTest.testReadLine
  testSys::BufTest.testReadAllLines
  testSys::BufTest.testEachLine
  testSys::BufTest.testReadAllStr
  testSys::BufTest.testPipe
  testSys::BufTest.testHex
  testSys::BufTest.testDigest
  testSys::CallTest.testNonVirtual
  testSys::CompilerJsTest.testMethodHiding
  testSys::EnvTest.testPlatform
  testSys::EnvTest.testDirs
  testSys::EnvTest.testMisc
  testSys::EnvTest.testFindFile
  testSys::EnvTest.testProps
  testSys::EnvTest.testLocale
  testSys::EnvTest.testIndex
  testSys::EnvTest.testShutdownHooks
  testSys::FileTest.testTestDir
  testSys::FileTest.testWalk
  testSys::FileTest.testNormalize
  testSys::FileTest.testCheckSlash
  testSys::FileTest.testCreateTemp
  testSys::FileTest.testCreateAndDelete
  testSys::FileTest.testCopyTo
  testSys::FileTest.testMoveTo
  testSys::FileTest.testStreamConvenience
  testSys::FileTest.testReadAllLinesNL
  testSys::FileTest.testModifyTime
  testSys::FileTest.testOsPath
  testSys::FileTest.testOpen
  testSys::FileTest.testMmap
  testSys::FileTest.testSync
  testSys::FloatTest.testLocale
  testSys::PodTest.testList
  testSys::ProcessTest.testStdMerged
  testSys::ProcessTest.testStdSeparate
  testSys::ProcessTest.testStrMerged
  testSys::ProcessTest.testStrSeparate
  testSys::ProcessTest.testNullOut
  testSys::ProcessTest.testIn
  testSys::ProcessTest.testEnv
  testSys::RegexTest.testSpecialChars
  testSys::StreamTest.testBasicIO
  testSys::StreamTest.testBufIO
  testSys::StreamTest.testBinary
  testSys::StreamTest.testCharUtf8
  testSys::StreamTest.testCharUtf16BE
  testSys::StreamTest.testCharUtf16LE
  testSys::StreamTest.testISO_8859
  testSys::StreamTest.testChar
  testSys::StreamTest.testUnread
  testSys::StreamTest.testReadChars
  testSys::StreamTest.testReadLine
  testSys::StreamTest.testReadStrToken
  testSys::StreamTest.testReadAllLines
  testSys::StreamTest.testEachLine
  testSys::StreamTest.testReadAllStr
  testSys::StreamTest.testPipe
  testSys::StrTest.testInterpolation
  testSys::StrTest.testLocale
  testSys::ZipTest.testOpen
  testSys::ZipTest.testCreate
***
*** 73 FAILURES [52 tests, 538 methods, 171310 verifies] 
***

Most of the causes is:

sys::IOErr: Cannot create dir: /temp/test
  fan.sys.LocalFile.createDir (LocalFile.java:264)
  fan.sys.LocalFile.create (LocalFile.java:222)
  fan.sys.Test.tempDir (Test.java:233)
  testSys::StreamTest.testCharUtf16BE (StreamTest.fan:419)
  java.lang.reflect.Method.invokeNative (Method.java)
  java.lang.reflect.Method.invoke (Method.java:521)
  fan.sys.Method.invoke (Method.java:558)
  fan.sys.Method$MethodFunc.callList (Method.java:204)
  fan.sys.Method.callList (Method.java:138)
  fanx.tools.Fant.runTest (Fant.java:174)
  fanx.tools.Fant.test (Fant.java:94)
  fanx.tools.Fant.test (Fant.java:30)
  fanx.tools.Fant.run (Fant.java:259)

go4 Sat 27 Jan 2018

Hi Brian, I make a new way to fix this. Follow the strategy B:

Disallow subclasses to reuse internal or private method names which would override a method in a parent class in the same pod (treat these as compiler errors and prevent it happening in first place)

change set

SlimerDude Sat 27 Jan 2018

This is also great, I'd love to see official Fantom on Android!

brian Sun 28 Jan 2018

Looking forward to seeing your designs on the other issues.

I can't remember why I said prevent in the same pod? Seems like you'd have to do for all pods, which in turn would break encapsulation and create a fragile base class problem. But that post is from seven years ago

go4 Sun 28 Jan 2018

The Fantom to Java protection modifiers mapping (from FTypeEmit.java line 418):

  • private => package-private
  • protected => public
  • internal => public

The private name is visible in the same pod and internal name is visible for all pods.

BTW: We nedd INVOKESPECIAL to invoke method of superclass's version and INVOKEVIRTUAL for others. There are already CallCtor CallMixin in opcode, so CallSuper would be more easy to distinguish from CallNoVirtual.

Login or Signup to reply.