LEAVU development phase, coding MPCD graphics

[Moa]

The Leavu2 Datalink host runs on my new 17" MacBook Pro. Nice work Yoda.

 

I'll try running the other programs and take a look at them.

 

Incidentally, the "Compile on Save" option in the project properties might speed up your development a little.

 

Edit: Leavu starts on OS X too, and client and server can talk.

Edited March 2, 2010 by Moa

[RvEYoda]

Jogl Already uses Java2d to render bitmapped text to textures.

 

http://people.eecs.ku.edu/~miller/Courses/JOGL/jogl-1.1.1-docs/com/sun/opengl/util/j2d/TextRenderer.html

 

I suggest you look into the TextRenderer source code. I have messed around with it a bit and made it a bit faster than the original,

but it still is by far the slowest part of the program. ( my performance numbers above are with my optimizations, otherwise it's even slower ).

I think we should rewrite the entire thing though.

 

I have tried GLJPanel before, and it was unfortunately significantly slower in some situations. A pretty frame is not important at the moment :)

 

Btw love ur scoring app design!

 

 

Implementing new parts

 

Moa maybe you could write a native JNI library for OSX and Linux that allows for keyboard detection?

Either that or we use some built in java stuff, but for windows i had to use JNI since I want to detect

keyboard input even when LEAVU2 does not have window focus.

 

It will be called by the Keyboard class in its state updating function :

 

   /**
    * Used by internal thread to update the exposed state of each subscribed key
    */
   private static final void scanKeyboard() {
       for (Key key : keysToScan) {
           boolean held = leavu.Win32.keyHeld(key.keyCode) < 0;
           key.hasBeenPressed = held && !key.isHeld;
           key.hasBeenReleased = !held && key.isHeld;
           key.isHeld = held;
       }
   }

 

The native call here is "leavu.Win32.keyHeld(int)"

In Windows I implemented the keyHeld function by using GetKeyState()

 

JNIEXPORT jshort JNICALL Java_leavu_Win32_keyHeld(JNIEnv *env, jclass cls, jint keyCode) {
return (jshort)GetKeyState((int)keyCode);
}

 

GLJPanel problems

 

When set up to try a GLJPanel now for performance testing I get the following issue when resizing the window,

so for now at least Ill stick with GLCanvas.

 

Some geometric shapes below are missing and Text is completely scrambled after resizing a GLJPanel. It may be

due to my TextRenderer Optimizations or my hacking of the Animator class.

Edited March 2, 2010 by =RvE=Yoda

[Boberro]

Yoda you are godlike :D

[Moa]

Jogl Already uses Java2d to render bitmapped text to textures.

 

http://people.eecs.ku.edu/~miller/Courses/JOGL/jogl-1.1.1-docs/com/sun/opengl/util/j2d/TextRenderer.html

 

Yes, but there are no guarantees as to what is cached and what is not. I was suggesting pre-rendering to BufferedImages and then simply copying those buffered images to your graphics context. That way you are only doing a bit-blit and then you are only fill-rate limited. This technique can be used for all the static text and images in the application (which is acually quite a lot, they don't change shape mauch, just position).

 

I suggest you look into the TextRenderer source code. I have messed around with it a bit and made it a bit faster than the original,

but it still is by far the slowest part of the program. ( my performance numbers above are with my optimizations, otherwise it's even slower ).

I think we should rewrite the entire thing though.

 

I have tried GLJPanel before, and it was unfortunately significantly slower in some situations. A pretty frame is not important at the moment :)

 

I'll be trying this out tonight when I get home from work.

 

Btw love ur scoring app design!

Thank you very much. With Swing you can use gradients and a whole bunch if existing effects to make everything look nice. Maybe using GLCanvas in a Swing app might be an good idea - plus it might better solve the layout issues.

 

 

Implementing new parts

 

Moa maybe you could write a native JNI library for OSX and Linux that allows for keyboard detection?

Either that or we use some built in java stuff, but for windows i had to use JNI since I want to detect

keyboard input even when LEAVU2 does not have window focus.

 

It will be called by the Keyboard class in its state updating function :

 

   /**
    * Used by internal thread to update the exposed state of each subscribed key
    */
   private static final void scanKeyboard() {
       for (Key key : keysToScan) {
           boolean held = leavu.Win32.keyHeld(key.keyCode) < 0;
           key.hasBeenPressed = held && !key.isHeld;
           key.hasBeenReleased = !held && key.isHeld;
           key.isHeld = held;
       }
   }

The native call here is "leavu.Win32.keyHeld(int)"

In Windows I implemented the keyHeld function by using GetKeyState()

 

JNIEXPORT jshort JNICALL Java_leavu_Win32_keyHeld(JNIEnv *env, jclass cls, jint keyCode) {
   return (jshort)GetKeyState((int)keyCode);
}

GLJPanel problems

 

When set up to try a GLJPanel now for performance testing I get the following issue when resizing the window,

so for now at least Ill stick with GLCanvas.

 

Some geometric shapes below are missing and Text is completely scrambled after resizing a GLJPanel. It may be

due to my TextRenderer Optimizations or my hacking of the Animator class.

 

I'll take a look. On Linux and Mac the window needs focus to get keyboard input, but that's ok since LockOn can't run directly on those platforms anyway. Better to try and use the facilities Java has rather than do JNI.

[Moa]

Oh yeah, what does the refresh rate of the application need to be? 30 Hz? 60 Hz? unlmited?

 

Seems inefficient to re-render information that is updated from Lua at around 10 Hz. Perhaps we could update a buffered display texture no more than 30 Hz. Would allow more screens to be rendered.

[RvEYoda]

Oh yeah, what does the refresh rate of the application need to be? 30 Hz? 60 Hz? unlmited?

 

Seems inefficient to re-render information that is updated from Lua at around 10 Hz. Perhaps we could update a buffered display texture no more than 30 Hz. Would allow more screens to be rendered.

 

I provide arbitrary data input rates from lockon (default is 30-50 fps), and render the displays at the same rate that data comes :). I can use leavu radar screen with no latency from the ingame radar screen at all.

 

There are 3 threads working.

 

Thread 1 is the data receiving thread. When it detects a new set of data received, it notifies...

Thread 2 updates the state of the instrments and then notifies the ...

Thread 3, rendering thread, renders a frame!

 

Basically Thread 1 does data receiving and pattern recognition, if the receive pattern is matched it

sends the received dat sequence for processing.

while (!toDie && (b = is.read()) != -1) {
   byte[] data = receiveBuffer.input((byte) b);
       if (data != null) {
            subscriber.process(data);
            Thread.yield();
       }
   }
}

 

where the read-command is set to a 1.5 seconds timeout. ( more breaks the loop )

The "process(data)" call notifies Thread 2, where each instruments receives and

processes the new data, and then decides what to place in its "rendering queue":

 

while (true) {
   try {
       synchronized (this) { wait(100); }
       for (Instrument instrument : instruments)
           instrument.update();
       animator.update();
   } catch (Exception e) {
       e.printStackTrace();
   }
}

 

and Thread 3 renders a frame with the current "rendering queue" of each instrument:

while (true) {
   try {
       synchronized (this) { wait(100); }
       for (GLCanvas screen: screens)
           screen.display();
       } catch (Exception e) {
           e.printStackTrace();
       }
   }
}

 

Most models are rendered using automatically cached GLlists. ( display lists )

The first time they are rendered they use the glVertex3d(...), and the second call

just uses the display list generated by call 1. If a previously generated display list

is not used for one frame, I remove it from OpenGL's internal state.

Edited March 3, 2010 by =RvE=Yoda

[Moa]

Thanks. I haven't looked over all the new code so some of my questions won't be relevant.

 

I can understand refreshing LockOn faster than 30 Hz but can't understand why mostly static text would need to be refreshed faster than this. You can refresh the model data at any speed, it's just that you might as well sync updates to the refresh of the display (whatever rate that is) or whatever rate humans would notice (which is probably 3ms for most LCDs that have persistence that limits all visible changes to be slower than this).

 

I'm just wondering if we run faster by eliminating unecessary work (that is one way to optimise, the other is to do things more effiicently but that usually reaches its limits sooner).

[RvEYoda]

Thanks. I haven't looked over all the new code so some of my questions won't be relevant.

 

I can understand refreshing LockOn faster than 30 Hz but can't understand why mostly static text would need to be refreshed faster than this. You can refresh the model data at any speed, it's just that you might as well sync updates to the refresh of the display (whatever rate that is) or whatever rate humans would notice (which is probably 3ms for most LCDs that have persistence that limits all visible changes to be slower than this).

 

I'm just wondering if we run faster by eliminating unecessary work (that is one way to optimise, the other is to do things more effiicently but that usually reaches its limits sooner).

 

You can chose yourself what rate the data is sent at.

The display only renders when data arrives, only when

something has changed. ( except for menu text )

 

Also if you want really smooth instruments, i

recommend 50 hz data input rate which generates

a completely smooth feeling imo. ( I run 30 though,

I find it decent, it is just a number in the export script )

Edited March 3, 2010 by =RvE=Yoda

[Moa]

Data rate can be anything. This is not the problem.

 

Do you think there ought to be an upper limit to the display refresh rate or should it be unlimited? ('No' is an ok answer, but 'yes' would be better)

[RvEYoda]

Data rate can be anything. This is not the problem.

 

Do you think there ought to be an upper limit to the display refresh rate or should it be unlimited? ('No' is an ok answer, but 'yes' would be better)

 

Why should there be?

If someone wants to update his instruments at 200 Hz, why not let him? :)

 

Now that I think of it, there is already an upper limit set. I limited the data export

rate to the framerate of lockon, so you cant go above that. Actually export depends

on the "beforeNextFrame" function in lua, so this guarantees there will not be any issues

with a lot of exports "queueing up" during a standard game stutter. The sim must produce

at least 1 frame before another data batch is sent to Leavu2.

 

Here is me feeding leavu2 at 30 Hz from FC2

 

Edited March 3, 2010 by =RvE=Yoda

[Moa]

Thanks for the vids. They are very impressive, especially the second one in dimmed light.

 

Ok, no upper limit on frame rate apart from that imposed by export.

[Moa]

Been profiling stuff and trying different options. I noticed that limiting the refresh rate to the monitor VSync rate dropped CPU usage a lot (which might get you more simultaneous instances).

 

The JoGL command to do this is:

gl.setSwapInterval(1);

 

I couldn't see it used in any of the four LEAVU2 NetBeans projects. So perhaps give that a go and see whether it does what you want.

 

Something else that can be tried is the use of an OpenGL profiler. I have not used one myself but I hear they're good for finding pipeline stalls and seeing whether you are fill-rate limited (which I suspect you may be given your already high frame rates and large number of rendering surfaces).

[RvEYoda]

Been profiling stuff and trying different options. I noticed that limiting the refresh rate to the monitor VSync rate dropped CPU usage a lot (which might get you more simultaneous instances).

 

 

Compared to what?

 

Im not sure I've explained it clearly enough.

There IS a limit to the upper frame rate rendered.

Without that the program would try to render at the highest possible fps (1000... ?).

But now it only renders a new frame when data arrives, which is capped by the lockon

framerate ( unlikely to go above 100?) and/or a custom minimum time diff between frames.

 

By default configured to what represents 30 fps, and would probably not be raised much higher than 60.

If you limit rendering to screen update rate (also 60 fps), you should not gain any performance?

I dont follow :(

 

Code below from my custom OpenGL Animator class.

while (true) {
   try {
       synchronized (this) { wait(100); } <-- Waits 100 ms or until notified ( by new data ) to render a single frame
       for (GLCanvas screen: screens)
           screen.display(); <-- Tells the openGL thread to render a frame
       } catch (Exception e) {
           e.printStackTrace();
       }
   }
}

 

I added gl.glsetSwapInterval(1);

to the init code of the Leavu2DRenderer class.

It isnt going to hurt I guess

 

 

Results

 

Well I don't understand this at all.

Im still rendering it at 30 fps fully smooth but the gl.glsetSwapInterval(1);

cut the total CPU usage of leavu2 by almost 40 % :P. Must be some internal

OpenGL thing! Thx Moa, can you explain why this helps even though I only

call the OpenGL rendering thread at 30 fps ?

It froze when I went above 67 mfds now. Shit I dont see why it should freeze :P

 

Commited it to the repository

 

update: The freeze when using a LOT of displays seems to be caused by Java AWT having problems with

adding the GLCanvas to the gridbag of the gui window. It is likely, that if we do not use AWT and move to

a 100% OpenGL based solution, that these problems would disappear.

 

 

New tests

 

I can add 200 GLCanvas objects no problem.

I can add 200 AWT windows no problem.

However if I add more than 40-50 GLCanvas objects each inside its own AWT Frame the program freezes,

even if I have turned off the actual rendering to the GLCanvas objects and they arent doing anything

This is also what one of my friends thought earlier..hmm. AWT and JOGL deadlock each other ? :P

My friend also thought Swing would be even worse for it

Edited March 3, 2010 by =RvE=Yoda

[RvEYoda]

Found the cause of the bug

 

It has nothing to do with LEAVU, the bug is caused by AWT-JOGL interactions.

I wrote the following test program and it shows the same crashes once

you create enough displays. This test program renders nothing, it just adds windows

when you press a button, until it freezes, and isnt using anything except standard

JOGL and AWT code.

 

package joglscreens;

import java.awt.Frame;
import javax.media.opengl.GLCanvas;

public class MainWindow extends javax.swing.JFrame {

   public MainWindow() {
       initComponents();
   }

   @SuppressWarnings("unchecked")
   private void initComponents() {
      ...*auto generated code for the window*...
   }             

   private void jButton2ActionPerformed(java.awt.event.ActionEvent evt) {                                         
       Frame F = new Frame();
       F.setSize(400, 400);
       F.add(new GLCanvas());
       F.setVisible(true);
   }                                        

   public static void main(String args[]) {
       java.awt.EventQueue.invokeLater(new Runnable() {
           public void run() {
               new MainWindow().setVisible(true);
           }
       });
   }

   private javax.swing.JButton jButton2; 

}

 

I have opened a thread on JOGL forums regarding this here:

http://www.javagaming.org/index.php/topic,22075.msg182135.html#msg182135

Edited March 3, 2010 by =RvE=Yoda

[topol-m]

Hey great video! Leavu looks amazing :thumbup:

[Moa]

Yoda it is not so much an internal JoGL/OpenGL thing, it is normal high performance programming.

 

Remember for smooth display we don't need a frame rate any higher than the monitor's display rate, we actually need *low latency*.

 

By telling JoGL to render at the monitor speed we're doing several things:

 

1) Not doing any more work than we absolutely have to (even if we did more a human cannot notice).

2) Telling JoGL that it's ok to let the rest of the system have some time too.

3) Avoiding tearing in the display when a refresh coincides with a vsync.

 

Of these, number 2 probably makes the most difference. If we hog all the CPU then the rest of the system can't run, and eventually the O/S decides that it's time to take over and takes control from your app and lets all the pent up tasks in the system run. In that case it can be inconvenient when the rest of the sytem runs and you miss a vsync update, which makes your frame rate drop to less intermittently. If the rest of the system gets plenty of chances to run (eg. clear I/O etc) then the chances are that when Leavu needs the CPU it'll get it.

 

I notice you do a thread yield in places, which is good, but is advisary only. It would be much better to use:

 

try {

Thread.sleep(1);

} catch (InterruptedException ex) {

// Ignore interruptions.

// nb. damn checked exceptions, they have their place, but this is not it!

}

Ok, with regard to locking. It may be a legitimate deadlock but more likely something is blocking the Event Dispatch Thread (EDT) - which is required by the Animator to call display(). I notice some of the code is EDT aware but some is not. If you are doing any Swing operation (eg. calling a JFrame method that creates any events) then you must use SwingUtilities.invokeAndWait or invokeLater

 

I don't think your friend is right. If the application was Swing then it would probably run more smoothly, not less. After Java .6.0_u10 *all* Swing rendering uses JoGL (or DirectX on Windows). If you are using JoGL on top of JoGL Swing then things get simpler for it (it doesn't have to lock and unlock native resources all over the place). Plus, as I said before Swing allows much nicer effects and resizes properly. Also JoGL was multi-threaded and was found to be slow due to lock contention, by making it single-threaded it was found to be faster, but between multiple instances we might still be suffering from contention a bit (not much we can do about it - unless you want to try doing a partially transparent full screen app that renders all 100 panels).

 

nb. FindBugs tool can help find possible thread problems (mostly unsynchronized resources). It is conveniently packaged for NetBeans at the following site (aren't you glad you're using NetBeans :)):

https://sqe.dev.java.net/

 

Also, I haven't checked the leavu2 code (I'm at work now) but using glOrtho projection might speed things up for the leavu 2D rendering, since it doesn't have to do perspective projection calculations. You may already be using it.

 

Tonight I'll be looking at loading images as textures and rendering them. If we can pre-render the background of each display (onscreen menu items etc) and then only render the selected one then we might get another little speedup.

Edited March 3, 2010 by Moa

[RvEYoda]

I have been talking to Michael Bien ( one of the JOGL creators ) and it looks like the freeze issue

is indeed JOGL-AWT related, and may be solved with JOGL 2.0

http://www.javagaming.org/index.php/topic,22075.msg182135.html#msg182135

If you check my sample you will notice the crash is demonstrated without any actual rendering

( or connection to the leavu project ) at all. It is just a short AWT app ( as in my previous post ).

 

 

In Leavu2 :

 

Displaying of graphics is already capped at monitor refresh rate. ( like you mentioned )

Vsync is already on to avoid tearing.

Furthermore rendering is also limited so it only renders when new data arrives.

 

Remember for smooth display we don't need a frame rate any higher than the monitor's display rate, we actually need *low latency*.

The rendering thread waits until a new data batch arrives, then renders a frame.

This frame is transmitted to the monitor intact asap ( with vsync ). An approach with a locked

update rate is not good and I've tried it. It suffers because lockon transmits data at rates depending on its

framerate. The latency will be poor because it is not synchronized with the data input rates.

 

In fact I found quite a bit of bugs in the integrated features of Netbeans. However when I tried Eclipse I found even

more issues so I went back to netbeans. The integrated SVN client of netbeans is complete crap though :P

 

If we are going to work together, then please, can you not considering using an IM software?

This forum posting, PMs, IDE chat is hopeless. It takes 2 weeks to complete 1 minute of discussing something :)

I really appreciate your input but this type of communication is simply hopeless.

Edited March 3, 2010 by =RvE=Yoda

[Moa]

IM yes, but due to me being almost exactly on the other side of the Earth I am at work while you are up. Since I'm consulting at a client site I can't get the IM out through their firewall.

 

I do jump on Leavu2 kenai though from time to time to see if you are there.

 

Using double buffered all we'll be doing when LockOn data arrives is updating the back buffer (burning CPU/GPU time) at a rate faster than the buffers are swapped.

[RvEYoda]

IM yes, but due to me being almost exactly on the other side of the Earth I am at work while you are up. Since I'm consulting at a client site I can't get the IM out through their firewall.

 

I do jump on Leavu2 kenai though from time to time to see if you are there.

 

Using double buffered all we'll be doing when LockOn data arrives is updating the back buffer (burning CPU/GPU time) at a rate faster than the buffers are swapped.

 

How it works right now.

The data batch rate can be capped ( by default 30-50 fps ).

 

Lockon data rate is not locked to an explicit frequency!

Instead it has a "minimum time before next update" and at least 1 lockon frame between updates.

This is due to a lesson learned in lrm development. Lrm originally used coroutines for a fixed update rate,

however this can cause major issues during normal game stutters ( the queued coroutine calls will then group

up and make the stutter MUCH worse). Later LRM switched to a "per beforeNextFrame" system, which is

also used by Leavu2 for data export. This has the drawback of not locking data export rate to a fixed

interval ( only a minimum or maximum interval ), but this is compensated by processing teh data upon arrival.

 

Lockon data export

function luaExportBeforeNextFrame
  if currentTime - lastExportTime > minimumInterval then
     lastExportTime = currentTime;
     ExportData();      
  end
end

Edited March 3, 2010 by =RvE=Yoda

[Moa]

Yes, I agree LockOn data rate should not not capped in any way (I must have been unclear about this).

 

What I was suggesting is that the "Data process and frame generation events" be limited to vsync. Doing it any faster makes no difference to screen, only to CPU usage.

 

That's great news about JOGL 2.0 fixing that locking problem. The other suggestion they made was to use one context - which I also suggested and can explain to you if you want (although managing the sub-windowing would be a pain).

[RvEYoda]

Yes, I agree LockOn data rate should not not capped in any way (I must have been unclear about this).

 

No, data export rate IS limited, I'm just saying it isnt an explicit frequency (time/second), but

a minimum time interval between data batches.

It should be capped, because I dont want to steal cpu time from lockon's single game thread,

nor export faster than our monitor's update rate :)

[Moa]

Copy.

[RvEYoda]

Copy.

 

Could you explain to me how the Thread.yield works and why Thread.sleep(1) is better?

Edited March 3, 2010 by =RvE=Yoda

[Moa]

Could you explain to me how the Thread.yield works and why Thread.sleep(1) is better?

I usually use yield in combination with sleep to explicitly let other threads run, is this not

right ?

 

I had some strange bluescreens when using multiple Thread.sleep(LowValue) instead of

Thread.yield.

 

My understanding is that yield is advisary only, and might not do anything, and if it does yield then the process continues to run and may immediately re-schedule the yielded thread. Thus, the CPU remains in use, although the thread within the process may change.

 

Sleep makes the thread sleep which relinquishes the CPU. The O/S scheduler may then allow other processes to run, or may reschedule the process if no one else needs it. The CPU does not remain in use by your program (letting other processes complete their tasks).

 

I wonder if the sleep() problem you observed is actually related to the locking problem you were seeing?

 

ps. I've been on kenai chat for a while :)

[RvEYoda]

Yes this is what I mean, Kenai chat is incredibly impractical. Much better to have a working IM,

even if it is only from home :).