Saturday, December 1, 2012

Shell: Summing up lots of (large) numbers

Sometimes you want to know the exact amount of bytes all the files in a directory tree takes. For example, checking the sum of sizes of all files is a quick way to see if a copy operation went OK - if they are the same, there are reasons to believe that it is OK.

du gives varying numbers from filesystem to filesystem

However, 'du' doesn't always give the right number - is it the size of the directory nodes causing differences?
du -sb .

awk-solutions have major precision problems

The solutions flying across the intertubes using awk, goes awry with large numbers. The following have a ceiling on 2147483647, as that is the max of a 32 bit integer number. Absurdly, awk just displays that if it reaches the limit.
find . -type f -printf "%s \n" | awk '{sum += $1 } END {printf "%i bytes\n",sum }'

You can get around that by going to floats (awk really uses double-precision floating-point for all numbers), but then you loose the whole point, exactness:
find . -type f -printf "%s \n" | awk '{sum += $1 } END { printf "%.1f bytes\n",sum }'

The Solution: bc - arbitrary precision calculator language!

Finding the sum of all the files in a directory tree:
echo `find . -type f -printf "%s+"`0 | bc

Same in GiB (GigaBytes as in 1024^3 bytes), using absurd scale to get exactness ('scale' is bc's concept of decimal precision) :
echo scale=50\; \(`find . -type f -printf "%s+"`'0)/(1024^3)' | bc

If you want MiB or KiB, then change the ^3 to ^2 or ^1 respectively.

Thursday, October 4, 2012

KD-101 smoke detector batteries

I have a six-unit setup of KD-101 Smoke Detectors - they are conveniently linked together via radio.These are OEM units, and thus come packaged in utterly different packages, one, two or three in a package, and can be bought at very different price points (I've found them for 3 for 200 NOK, to 1 for 300!!), so shop around.

Each unit uses both a 9 volt battery and three AA (LR03) 1,5 volt batteries. I wondered what the different batteries were used for; Apparently the unit functions just fine with only the 9v battery - it both runs the Test OK, and it triggers the other alarms just fine. However, it apparently cannot receive a trigger from the other alarms just on the 9v battery - so apparently, the 3 x 1,5v batteries are used to run the receiver radio.

In my (unsuccessful) search of this answer found some instruction manuals, a good one is here (PDF, Norwegian), and another one here (PDF). None told what the different batteries were for, so that's the reason for this post.

When the alarm beeps once every 45 seconds, the 3x AA batteries is to be changed.
When it beeps once every 60 seconds, the 9v battery must be changed. (This difference in beep-timing is according to the Norwegian manual)

I just changed all batteries to use Lithium cells. These typically last 10 years in a smoke detector, and the detector states that it should be exchanged after about 8 years, so I guess we're talking "life-time batteries" then! It cost just short of a 1000 kroners to buy 6 9v and 4x4 1,5 AA batteries - I've never been in the vicinity of paying that much for batteries before - not even car and boat batteries are than expensive! And it's more than I paid for the detectors themselves..!

Wednesday, May 5, 2010

Linux Java Repeats RELEASED KeyEvents

In Java on Linux, there is a 12 year old bug in handling of keyboard auto-repeat. As on Windows, both KEY_PRESSED and KEY_TYPED repeats, but on Linux, also KEY_RELEASED repeats, while on Windows the released-event comes only when the user releases the key.

This class can be installed as an AWTEventListener, and will seemingly fix this. Note that the class can be installed on both Windows and Linux - it won't affect already correct behavior.

package com.example;

import java.awt.AWTEvent;
import java.awt.Component;
import java.awt.EventQueue;
import java.awt.Toolkit;
import java.awt.event.AWTEventListener;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.event.KeyEvent;
import java.util.HashMap;
import java.util.Map;

import javax.swing.Timer;

/**
 * This {@link AWTEventListener} tries to work around a 12 yo
 * bug in the Linux KeyEvent handling for keyboard repeat. Linux apparently implements repeating keypresses by
 * repeating both the {@link KeyEvent#KEY_PRESSED} and {@link KeyEvent#KEY_RELEASED}, while on Windows, one only
 * gets repeating PRESSES, and then a final RELEASE when the key is released. The Windows way is obviously much more
 * useful, as one then can easily distinguish between a user holding a key pressed, and a user hammering away on the
 * key.
 * 
 * This class is an {@link AWTEventListener} that should be installed as the application's first ever
 * {@link AWTEventListener} using the following code, but it is simpler to invoke {@link #install() install(new
 * instance)}:
 * 
 * 
 * Toolkit.getDefaultToolkit().addAWTEventListener(new {@link RepeatingReleasedEventsFixer}, AWTEvent.KEY_EVENT_MASK);
 * 
* * Remember to remove it and any other installed {@link AWTEventListener} if your application have some "reboot" * functionality that can potentially install it again - or else you'll end up with multiple instances, which isn't too * hot. * * Notice: Read up on the {@link Reposted} interface if you have other AWTEventListeners that resends KeyEvents * (as this one does) - or else we'll get the event back. *

Mode of operation

* The class makes use of the fact that the subsequent PRESSED event comes right after the RELEASED event - one thus * have a sequence like this: * *
 * PRESSED 
 * -wait between key repeats-
 * RELEASED
 * PRESSED 
 * -wait between key repeats-
 * RELEASED
 * PRESSED
 * etc.
 * 
* * A timer is started when receiving a RELEASED event, and if a PRESSED comes soon afterwards, the RELEASED is dropped * (consumed) - while if the timer times out, the event is reposted and thus becomes the final, wanted RELEASED that * denotes that the key actually was released. * * Inspired by http://www.arco.in-berlin.de/keyevent.html * * @author Endre Stølsvik */ public class RepeatingReleasedEventsFixer implements AWTEventListener { private final Map _map = new HashMap(); public void install() { Toolkit.getDefaultToolkit().addAWTEventListener(this, AWTEvent.KEY_EVENT_MASK); } public void remove() { Toolkit.getDefaultToolkit().removeAWTEventListener(this); } @Override public void eventDispatched(AWTEvent event) { assert event instanceof KeyEvent : "Shall only listen to KeyEvents, so no other events shall come here"; assert assertEDT(); // REMEMBER THAT THIS IS SINGLE THREADED, so no need for synch. // ?: Is this one of our synthetic RELEASED events? if (event instanceof Reposted) { // -> Yes, so we shalln't process it again. return; } // ?: KEY_TYPED event? (We're only interested in KEY_PRESSED and KEY_RELEASED). if (event.getID() == KeyEvent.KEY_TYPED) { // -> Yes, TYPED, don't process. return; } final KeyEvent keyEvent = (KeyEvent) event; // ?: Is this already consumed? // (Note how events are passed on to all AWTEventListeners even though a previous one consumed it) if (keyEvent.isConsumed()) { return; } // ?: Is this RELEASED? (the problem we're trying to fix!) if (keyEvent.getID() == KeyEvent.KEY_RELEASED) { // -> Yes, so stick in wait /** * Really just wait until "immediately", as the point is that the subsequent PRESSED shall already have been * posted on the event queue, and shall thus be the direct next event no matter which events are posted * afterwards. The code with the ReleasedAction handles if the Timer thread actually fires the action due to * lags, by cancelling the action itself upon the PRESSED. */ final Timer timer = new Timer(2, null); ReleasedAction action = new ReleasedAction(keyEvent, timer); timer.addActionListener(action); timer.start(); _map.put(Integer.valueOf(keyEvent.getKeyCode()), action); // Consume the original keyEvent.consume(); } else if (keyEvent.getID() == KeyEvent.KEY_PRESSED) { // Remember that this is single threaded (EDT), so we can't have races. ReleasedAction action = _map.remove(Integer.valueOf(keyEvent.getKeyCode())); // ?: Do we have a corresponding RELEASED waiting? if (action != null) { // -> Yes, so dump it action.cancel(); } // System.out.println("PRESSED: [" + keyEvent + "]"); } else { throw new AssertionError("All IDs should be covered."); } } /** * The ActionListener that posts the RELEASED {@link RepostedKeyEvent} if the {@link Timer} times out (and hence the * repeat-action was over). */ private class ReleasedAction implements ActionListener { private final KeyEvent _originalKeyEvent; private Timer _timer; ReleasedAction(KeyEvent originalReleased, Timer timer) { _timer = timer; _originalKeyEvent = originalReleased; } void cancel() { assert assertEDT(); _timer.stop(); _timer = null; _map.remove(Integer.valueOf(_originalKeyEvent.getKeyCode())); } @Override public void actionPerformed(@SuppressWarnings ("unused") ActionEvent e) { assert assertEDT(); // ?: Are we already cancelled? // (Judging by Timer and TimerQueue code, we can theoretically be raced to be posted onto EDT by TimerQueue, // due to some lag, unfair scheduling) if (_timer == null) { // -> Yes, so don't post the new RELEASED event. return; } // Stop Timer and clean. cancel(); // Creating new KeyEvent (we've consumed the original). KeyEvent newEvent = new RepostedKeyEvent((Component) _originalKeyEvent.getSource(), _originalKeyEvent.getID(), _originalKeyEvent.getWhen(), _originalKeyEvent.getModifiers(), _originalKeyEvent.getKeyCode(), _originalKeyEvent.getKeyChar(), _originalKeyEvent.getKeyLocation()); // Posting to EventQueue. Toolkit.getDefaultToolkit().getSystemEventQueue().postEvent(newEvent); // System.out.println("Posted synthetic RELEASED [" + newEvent + "]."); } } /** * Marker interface that denotes that the {@link KeyEvent} in question is reposted from some * {@link AWTEventListener}, including this. It denotes that the event shall not be "hack processed" by this class * again. (The problem is that it is not possible to state "inject this event from this point in the pipeline" - one * have to inject it to the event queue directly, thus it will come through this {@link AWTEventListener} too. */ public interface Reposted { // marker } /** * Dead simple extension of {@link KeyEvent} that implements {@link Reposted}. */ public static class RepostedKeyEvent extends KeyEvent implements Reposted { public RepostedKeyEvent(@SuppressWarnings ("hiding") Component source, @SuppressWarnings ("hiding") int id, long when, int modifiers, int keyCode, char keyChar, int keyLocation) { super(source, id, when, modifiers, keyCode, keyChar, keyLocation); } } private static boolean assertEDT() { if (!EventQueue.isDispatchThread()) { throw new AssertionError("Not EDT, but [" + Thread.currentThread() + "]."); } return true; } } package com.example; import java.awt.EventQueue; import java.awt.FlowLayout; import java.awt.event.KeyEvent; import java.awt.event.KeyListener; import javax.swing.JButton; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.JPanel; import javax.swing.WindowConstants; /** * Tester for {@link RepeatingReleasedEventsFixer}. * * @author Endre Stølsvik */ public class XRepeatingReleasedEventsFixer { public static void main(String[] args) { EventQueue.invokeLater(new Runnable() { @Override public void run() { startGUI(); } }); } private static void startGUI() { new RepeatingReleasedEventsFixer().install(); JFrame frame = new JFrame("TestFrame"); JPanel main = new JPanel(new FlowLayout()); JButton listenedButton = new JButton("Have KeyListener"); main.add(listenedButton); listenedButton.addKeyListener(new KeyListener() { @Override public void keyPressed(KeyEvent e) { System.out.println("keyPressed: [" + e + "]."); } @Override public void keyTyped(KeyEvent e) { System.out.println("keyTyped: [" + e + "]."); } @Override public void keyReleased(KeyEvent e) { System.out.println("keyReleased: [" + e + "]."); } }); main.add(new JButton("No Listeners")); main.add(new JLabel("Try arrows, Ctrl, and chars,")); main.add(new JLabel("as well as multiple at once.")); frame.add(main); frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE); frame.setSize(260, 140); frame.setVisible(true); } }

Monday, March 1, 2010

Blog This! : Communicate - Google Chrome Help

Testing, 1, 2, 3 - Google's Blog This!

There's a bunch of official Google Chrome extensions for Google products - and this post is created using ..

Blog This! : Communicate - Google Chrome Help: "Post to Blogger with just one click"

This extension adds a Blog This! button to the toolbar that you can click to create a new Blogger post. The new post is pre-populated with a link to the web page you're on, as well as any text you've highlighted on that page. Edit the post to your liking and post it instantly to your blog! Learn more about Blogger

To learn more about the extension, visit its homepage in the Extensions Gallery. You can discover even more extensions in the gallery.

Thursday, February 18, 2010

JTextArea vs. Tab focus cycle

When inside a JTextArea, the Tab key inserts a tab character instead of going to the next field. The reason is obvious - but in several situations it is not desirable, in particular when it would not make sense to insert tab characters, for example in a text area meant for editing keywords.

Lets just spill the solution right away:
JTextArea text = new JTextArea();
text.setFocusTraversalKeys(KeyboardFocusManager.FORWARD_TRAVERSAL_KEYS, null);
text.setFocusTraversalKeys(KeyboardFocusManager.BACKWARD_TRAVERSAL_KEYS, null);

There is also a built-in solution, which is that Ctrl-[Shift-]Tab still works. Actually, those keys always work. Those keys are also the solution when you suddenly find yourself focus-stuck within a JTable. I first thought that a JTable constituted a a "focus cycle root", but it doesn't - it is just the same effect as with the JTextArea problem here outlined (and can be "fixed" the same way).

However, Ctrl-Tag does not quite give the feeling one typically is after, and in particular not when one use the JTextArea more as a solution to text that typically is less than a line big, but can become multiline. Such a field thus looks like a JTextField, but the Tab character suddenly have a completely different meaning.

I Googled a lot, and there are lots of solutions (or rather hacks), but didn't find the above solution, which seems to be the most straightforward after Java 1.5. The huge heap of questions and solutions should, IMHO, give a hint to the Swing team that a specific method for a JTextArea to not suck up Tab events should be made available.

The null means that it will take the focus keys from its parent.

Thursday, February 11, 2010

Anonymous class implementing interface

Why is this not allowed:
JPanel panel = new JPanel(...) implements Scrollable {
    ...
};

I've ended up trying to write something like that several times - and each time I start to wonder what is wrong! A problem is of course that you could not directly access the methods of the implemented interface, but you could still have instanceof'ed and cast'ed it. Some interfaces are also just used for marking, e.g. Cloneable.

It is allowed with a local class:
class ScrollableJPanel extends JPanel implements Scrollable {
    ScrollableJPanel() {
        super(...);
    }
    ...
}
JPanel panel = new ScrollableJPanel();
The restriction seems arbitrary.

PS: Oh my god Blogger sucks. I have GOT to get away from this crappy blogging software. How is it possible to not have made any progress the last 5 years?!

Saturday, January 30, 2010

Linux Java Thread Priorities workaround

First and foremost: This workaround was found by Akshaal, and his blogentry about the problem and the "fix" is here (2008-04-28).

Secondly: Fix/Workaround to let Java on Linux work somewhat with Thread priorities, start the JVM with the following argument:
-XX:ThreadPriorityPolicy=42

For some annoying reason, Sun has decided that to run with threads with working thread priorities on Linux, you have to be root. The logic behind this decision, is that to heighten thread priorities, you have to be root. But, says many of us, could you not just let us lower thread priorities, at least? No, says Sun. I believe they just don't quite understand what is requested.

The sun bug 4813310 tracks this, but it is Closed - Fix Delivered. So the chances of a fix are slim, unless some JDK developer takes mercy.

Anyway - Akshaal found out that if you just set the "ThreadPriorityPolicy" to something else than the legal values 0 or 1, say 2 or 42, or 666 as he suggest himself, a slight logic bug in Sun's JVM code kicks in, and thus sets the policy to be as if running with root - thus you get exactly what one desire. The operating system, Linux, won't allow priorities to be heightened above "Normal" (negative nice value), and thus just ignores those requests (setting it to normal instead, nice value 0) - but it lets through the requests to set it lower (setting the nice value to some positive value).

I wrote a little program to test out priorities and these flags. There are two obvious ways to test such scheduling: Either have a fixed amount of work for each thread at each priority, and time how long time it takes to execute this - or let the system run for a fixed time, and then count how much work is done during this time. I first thought of the latter, so I coded that - and as I think about it further, I realize that this must be the most correct check too: Since if the high priority jobs finish earlier, the timing data of the lower priority jobs will not be entirely correct: There are now, obviously, fewer threads competing for the CPU (of course, one could let each thread do X work, then note how long time that took - and then have the same thread do unlimited amount of work afterward, so that it didn't skew the measurements by exiting before the least priority thread was finished).


Here are the results (2010-01-29 on JDK 1.6.0_18):

As user, with the following arguments (for the rest of the runs, the things changing are user vs. root, and the ThreadPriorityPolicy):

-XX:ThreadPriorityPolicy=0
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintCompilation
  1       java.util.concurrent.atomic.AtomicLong::get (5 bytes)
--- n sun.misc.Unsafe::compareAndSwapLong
--- n sun.misc.Unsafe::compareAndSwapLong
2 java.util.concurrent.atomic.AtomicLong::compareAndSet (13 bytes)
3 java.util.Random::next (47 bytes)
4 java.util.Random::nextDouble (24 bytes)
1% com.example.XThreadPriorities$Runner::run @ 13 (70 bytes)
5 com.example.XThreadPriorities$Runner::run (70 bytes)
Warmup complete.
Running test.
Thread MIN_PRIORITY:[1], NORM_PRIORITY:[5], MAX_PRIORITY:[10].
3924841464 - Pri:1 - 448333661 451444766 472740484 413017157 417892801 548408461 605271612 567732522
3948617160 - Pri:2 - 592733363 404123318 425058572 604107501 519520686 418888175 526486630 457698915
4266211603 - Pri:3 - 761499319 427767980 429864266 603262254 560988584 418244133 641052397 423532670
3646125561 - Pri:4 - 404603024 416434536 429997217 431405433 641421096 478439739 420713663 423110853
3968532194 - Pri:5 - 415064763 422632368 419370153 604712758 561722145 418841260 521415092 604773655
4006791532 - Pri:6 - 420221100 422553508 614947805 561412057 641625815 479341536 432081606 434608105
4014589807 - Pri:7 - 585756485 418316991 520996504 474136186 637166908 419697931 434488836 524029966
4192942528 - Pri:8 - 480067579 472195975 589334074 519719204 640510132 420580090 462976949 607558525
3965592952 - Pri:9 - 419045425 475155351 473396403 540690307 520495432 522822449 598591571 415396014
3603044034 - Pri:10 - 419117989 421450664 520872436 429775340 566700552 418128806 414890139 412108108
TotalyDummy: [0.13069701389705252].
Heap
PSYoungGen total 27904K, used 957K [0x00000000e0e00000, 0x00000000e2d20000, 0x0000000100000000)
eden space 23936K, 4% used [0x00000000e0e00000,0x00000000e0eef640,0x00000000e2560000)
from space 3968K, 0% used [0x00000000e2940000,0x00000000e2940000,0x00000000e2d20000)
to space 3968K, 0% used [0x00000000e2560000,0x00000000e2560000,0x00000000e2940000)
PSOldGen total 63744K, used 0K [0x00000000a2a00000, 0x00000000a6840000, 0x00000000e0e00000)
object space 63744K, 0% used [0x00000000a2a00000,0x00000000a2a00000,0x00000000a6840000)
PSPermGen total 21248K, used 2534K [0x000000009d600000, 0x000000009eac0000, 0x00000000a2a00000)
object space 21248K, 11% used [0x000000009d600000,0x000000009d8799c8,0x000000009eac0000)

As user, with -XX:ThreadPriorityPolicy=1. Notice the warning:

Java HotSpot(TM) 64-Bit Server VM warning: -XX:ThreadPriorityPolicy requires root privilege on Linux
...
4074355867 - Pri:1 - 527880034 510291794 506778321 500873083 507354362 494273347 543124483 483780443
4070970485 - Pri:2 - 502093074 507450387 513538019 548011008 515777472 510318658 492394477 481387390
4166089696 - Pri:3 - 545072864 510212654 503618003 545454609 509985968 499563619 542154550 510027429
3938795103 - Pri:4 - 499309103 482262429 483892136 484047646 493485804 497920357 486252326 511625302
4030810973 - Pri:5 - 492208426 510091021 472609917 551737964 511926521 488736661 495151696 508348767
3991986357 - Pri:6 - 499139701 484193553 508453245 494823573 505428815 491609383 495135614 513202473
3970332529 - Pri:7 - 508806026 484872150 460815795 545766805 487437289 484429061 511512703 486692700
3985129481 - Pri:8 - 508044296 493994479 486757819 504809264 504582073 489230988 485472873 512237689
3993531405 - Pri:9 - 485761906 508066620 498470220 526708671 484472110 474413825 498665020 516973033
3880591568 - Pri:10 - 482605449 492966477 494274413 512614969 525299924 401406096 483149581 488274659


As root, with -XX:ThreadPriorityPolicy=1:
...
1192628609 - Pri:1 - 146366375 145886054 145439504 148594629 154873659 145337314 155007218 151123856
1600638473 - Pri:2 - 293845127 182254047 179729049 186290117 174455802 186991284 194082370 202990677
1865691639 - Pri:3 - 252269194 240399347 219981901 223897910 228422404 242806673 239024804 218889406
2405100112 - Pri:4 - 303976233 283413460 310125261 300915820 286445060 313188957 295836808 311198513
2895152789 - Pri:5 - 355221596 377247451 374614383 338382343 361193338 352556243 376620158 359317277
3575027949 - Pri:6 - 430714357 452538422 438269143 451133734 476982979 437064495 436451954 451872865
4443379518 - Pri:7 - 576373599 575729812 528069437 559187021 582029237 539218400 520491072 562280940
5573457006 - Pri:8 - 684559763 685997740 722733003 672837267 683740096 716802117 697600305 709186715
7105117744 - Pri:9 - 885538364 920253981 855196886 825353759 909130110 960094293 864861817 884688534
9047448786 - Pri:10 - 1102826483 1079059728 1170567845 1062711472 1103250654 1194296507 1152074721 1182661376


As user, with "hack" -XX:ThreadPriorityPolicy=42:
...
1908592461 - Pri:1 - 252347610 268364950 217692707 199375572 241284341 243700244 267888904 217938133
2553635695 - Pri:2 - 345500976 320245701 313000117 330487363 307702106 304209714 312839609 319650109
3061308188 - Pri:3 - 352952639 431153057 415115944 336767623 412447023 393943268 380318370 338610264
4023325349 - Pri:4 - 532822164 500246668 509185449 515950251 422229930 551608063 511732302 479550522
4789209609 - Pri:5 - 625601672 530227747 622740566 599139605 725003140 602205329 596298732 487992818
4730674281 - Pri:6 - 625476539 482470376 726537575 655716492 487662808 621938212 601082560 529789719
4881113871 - Pri:7 - 652977961 586461749 726919389 595618652 484349203 661139324 684148172 489499421
4533403095 - Pri:8 - 614634085 484084154 526523754 643123946 599005063 647009071 532566904 486456118
4727219915 - Pri:9 - 625026801 596318117 726547378 598978337 601971731 489284222 491219584 597873745
4827951033 - Pri:10 - 678619163 525617294 592417838 657638409 722179525 484460205 531506557 635512042



As user, with "hack" and specific setting of OS priorities. We start at Unix nice level 0 for the highest java priority, going down to 9 for the lowest java priority. Notice how this leads to the situation where a NORMAL priority java thread have a quite significant nice level (low priority) on the OS side. Thus, in a contended environment with other processes running on the machine competing for CPU resources, "normal" java threads will loose out to the other processes, so this is not a complete solution:

-XX:ThreadPriorityPolicy=42
-XX:JavaPriority10_To_OSPriority=0
-XX:JavaPriority9_To_OSPriority=1
-XX:JavaPriority8_To_OSPriority=2
-XX:JavaPriority7_To_OSPriority=3
-XX:JavaPriority6_To_OSPriority=4
-XX:JavaPriority5_To_OSPriority=5
-XX:JavaPriority4_To_OSPriority=6
-XX:JavaPriority3_To_OSPriority=7
-XX:JavaPriority2_To_OSPriority=8
-XX:JavaPriority1_To_OSPriority=9
...
1181594447 - Pri:1 - 156646712 163727007 149271484 148003186 146046042 142171936 136636764 139091316
1467758385 - Pri:2 - 181741621 207823162 180440610 178282147 181600823 175446181 174811880 187611961
1835430827 - Pri:3 - 249361299 241085322 198475979 220310800 212862944 227538859 223721751 262073873
2310513758 - Pri:4 - 298831212 311762873 263149604 261915158 279594960 280329975 284289057 330640919
2820606635 - Pri:5 - 417528424 337381991 374884808 300077072 359420239 346286405 382398636 302629060
3517176583 - Pri:6 - 450919929 431990553 434093985 473421346 471673385 375806611 457967591 421303183
4386636298 - Pri:7 - 526555707 475575838 600301353 603713184 570267602 475066321 489751343 645404950
5646571395 - Pri:8 - 897622033 693133727 703208339 759090217 712003197 591393673 618879272 671240937
6260518137 - Pri:9 - 797127673 797467534 739969033 913730558 735004475 798095374 736360300 742763190
10144752105 - Pri:10 - 1504673551 1155125325 1123163868 1302155597 1390019512 1216611268 1208440124 1244562860

To directly see the problem where normal threads, which have java priority 5, gets unfair treatment with the above solution, I ran a quick test with two such tests simultaneously using the following little shell script (I also didn't run for 5 minutes here):
$ cat test.sh 
#!/bin/sh

java \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCompilation -classpath eclipsebuild com.example.XThreadPriorities &

java \
-XX:ThreadPriorityPolicy=42 \
-XX:JavaPriority10_To_OSPriority=0 \
-XX:JavaPriority9_To_OSPriority=1 \
-XX:JavaPriority8_To_OSPriority=2 \
-XX:JavaPriority7_To_OSPriority=3 \
-XX:JavaPriority6_To_OSPriority=4 \
-XX:JavaPriority5_To_OSPriority=5 \
-XX:JavaPriority4_To_OSPriority=6 \
-XX:JavaPriority3_To_OSPriority=7 \
-XX:JavaPriority2_To_OSPriority=8 \
-XX:JavaPriority1_To_OSPriority=9 \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCompilation -classpath eclipsebuild com.example.XThreadPriorities &

This lead to the following result:
70479632 - Pri:1 -  15781306 7639485 8364936 7772256 7424872 7930535 7750748 7815494
80416568 - Pri:2 - 11971141 9843537 9242104 9872974 10394428 9485233 9989330 9617821
103451737 - Pri:3 - 15886246 12949826 11725092 13793650 12614427 11656444 12344883 12481169
143220837 - Pri:4 - 32940102 16299182 14991785 16375282 15634409 14790916 15922255 16266906
159230735 - Pri:5 - 20471865 19681922 18727162 25211867 17791552 18787357 19537079 19021931
199052863 - Pri:6 - 24375945 25213341 24205803 31016513 22616403 23760056 23570549 24294253
245144874 - Pri:7 - 32218827 32270732 30006087 31789704 27064661 32474676 30256065 29064122
318656402 - Pri:8 - 38021055 39259080 44766347 49071457 34923402 35975708 40544736 36094617
401223243 - Pri:9 - 48600625 49467360 60141538 49625176 43789040 56507123 48418800 44673581
483842551 - Pri:10 - 58150216 58609942 77028357 59819032 53866188 59828368 59407580 57132868
TotalWork: [2204719442] (TotalyDummy: [29.703316307452084]).

469978381 - Pri:1 - 78278984 54629208 68421300 50861462 59341367 47883726 52597471 57964863
446759823 - Pri:2 - 77297533 50977670 73351930 50608554 47847622 46878431 53587386 46210697
472936111 - Pri:3 - 59336726 53398323 72483088 72872361 62524544 50161550 54083565 48075954
460567710 - Pri:4 - 59351070 51692155 72606512 72652587 47014898 52273533 46462284 58514671
467930078 - Pri:5 - 58932415 51900432 71509566 73609344 59049535 51626557 46980275 54321954
416276726 - Pri:6 - 58922845 55446857 47075994 47381016 47463150 52581761 52572845 54832258
461001441 - Pri:7 - 69723843 59707979 47522091 53322726 47818110 67710544 60742003 54454145
455315271 - Pri:8 - 49172227 61493744 62323717 50959541 47825115 70684358 51816372 61040197
424012117 - Pri:9 - 49651798 56514868 46221793 54045253 53094738 52019573 55008343 57455751
428007514 - Pri:10 - 48523372 47959147 49071583 62248855 51265246 51405853 54465519 63067939
TotalWork: [4502785172] (TotalyDummy: [-14.989963287910708]).


The best solution for running as user is probably the "-XX:ThreadPriorityPolicy=42" without the specific setting of OS levels. In that scenario, a normal priority java thread gets a normal OS nice level, while it still is possible to let e.g. "batch processing threads" get a lower nice level. The only thing that I'd appreciate was if the (AWT) Event Dispatch Thread got a notch higher priority than all other threads. Java does exactly that, by setting the EDT's priority to 6 (but as shown, that doesn't matter here, all priorities higher than 5 gets effectively 5 anyway). However, by setting all other threads in a GUI application down a notch (to 4 or less), you'd get the same effect.

Just to point it out: The above solution should be the default. This gives at least a somewhat reasonable logic: You can lower the priority, but you cannot jack it above normal. What one really would want, is that these priorities only worked within the process, not globally across the system. Then one could jack some threads "above normal", as they would only affect the scheduling within this process' threads. But this would apparently require a change in Linux.

Here's the code. It should be possible to just copy-paste it onto a project node of a Eclipse project (stand on the project node, hit Ctrl-V), thus getting it runnable instantly:

package com.example;

import java.util.Random;

/**
* Tests the Thread priorities of a Java program by letting threads run "unlimited amounts" of work for a fixed period
* of time, counting how much work they get done.
*
* @author Endre Stølsvik
*/
public class XThreadPriorities {

static volatile boolean _runThreads;

public static final int PARALLELS = Runtime.getRuntime().availableProcessors();

public static final int RUNNING_WARMUP_SECONDS = 4;

public static final int RUNNING_SECONDS = 60 * 5;

public static void main(String[] args) throws InterruptedException {
// Run warmup
run(RUNNING_WARMUP_SECONDS);
System.out.println("Warmup complete.");
Thread.sleep(500);
System.out.println("Running test.");
// Run test
Runner[][] runners = run(RUNNING_SECONDS);

// Dump results.
System.out.println("Thread MIN_PRIORITY:[" + Thread.MIN_PRIORITY + "], NORM_PRIORITY:[" + Thread.NORM_PRIORITY
+ "], MAX_PRIORITY:[" + Thread.MAX_PRIORITY + "].");

double totalDummy = 0;
long totalWork = 0;
for (int j = 0; j < runners[0].length; j++) {
int pri = runners[0][j]._priority;
String msg = "Pri:" + pri + " - ";
long total = 0;
for (int i = 0; i < runners.length; i++) {
totalDummy = runners[i][j]._dummy;
assert pri == runners[i][j]._priority;
long counter = runners[i][j]._counter;
total += counter;
msg += " " + counter;
}
totalWork += total;
System.out.println(total + " - " + msg);
}
System.out.println("TotalWork: [" + totalWork + "] (TotalyDummy: [" + totalDummy + "]).");
}

private static Runner[][] run(int seconds) throws InterruptedException {
_runThreads = true;
int priorities = Thread.MAX_PRIORITY - Thread.MIN_PRIORITY + 1;
Runner[][] runners = new Runner[PARALLELS][priorities];
// Make threads
for (int i = 0; i < PARALLELS; i++) {
int pri = Thread.MIN_PRIORITY;
for (int j = 0; j < priorities; j++) {
runners[i][j] = new Runner(pri++);
}
}
// Start threads.
for (int i = 0; i < PARALLELS; i++) {
for (int j = 0; j < priorities; j++) {
runners[i][j].start();
}
}
// Run all threads for whatever number of seconds.
Thread.sleep(seconds * 1000);
// Ask threads to stop.
_runThreads = false;
// Make sure they're stopped by joining them
for (int i = 0; i < PARALLELS; i++) {
for (int j = 0; j < priorities; j++) {
runners[i][j].join();
}
}
return runners;
}

private static class Runner extends Thread {
final Random _random = new Random();
final int _priority;

public Runner(int priority) {
this._priority = priority;
}

long _counter = 0;
double _dummy = 0;

@Override
public void run() {
Thread.currentThread().setPriority(_priority);
while (_runThreads) {
_counter++;
_dummy += (_random.nextDouble() * 2 - 1) / 123.579 * (_random.nextDouble() * 2 - 1);
}
}
}
}


Saturday, January 9, 2010

Eclipse on Ubuntu Karmic behaves weirdly

I am installing my new machine these days, running Ubuntu Karmic Koala 9.10. I finally came to setting up my environment on Eclipse, going to install Subversive for SVN access.

I then find that the Install new software... dialog behaves very weird - I can't seem to get anything up in the checkable list box when I select anything from the "Work with:" site selector dropdown.

Also, I find that I very often am unable to click buttons on any dialog window - this also goes for the Preferences dialog (it is possible to Tab around and hit enter, though).

It turns out that both these problem is an incompatibility between SWT (the Standard Widget Toolkit that Eclipse uses) and GTK+ (The GIMP Toolkit, used by GNOME) v.2.18 and apparently also Compiz (The advanced compositing window manager that can be used with GNOME). The underlying problem is apparently that GTK has changed some internal way windows are handled that exposes not-quite-correct usage by some applications, notably SWT. So SWT have to fix it, which they have.

However, a fix is not "out" yet, but there is a workaround, which is to run..

export GDK_NATIVE_WINDOWS=true

..before invoking eclipse (in a terminal, obviously). Some people apparently find that if you reboot eclipse (restart, or switch workspace), the fix doesn't seem to stick - however it seems to for me.

Links to bugs: Eclipse, Ubuntu

Btw, same thing goes for Azureus (and hence Vuze), as they also use SWT.

Btw2, the same thing also goes for Flash in the browsers (typically one find this out by buttons in youtube videos not working!), but here google will have to be your friend.

Btw3, to install Subversive for subversion access, launch the Install new software... dialog, select the Galileo site from the dropdown, and just search for "subversive". Find and check Subversive SVN Team Provider (Incubation) under Collaboration. Also check the Mylyn Integration if applicable, then install. When you reboot, the Subversive plugin will find out that you have not installed those annoying Subversive Connectors (the actual subversion handlers - which are not distributed from the eclipse main site due to some licensing bullshit) which you had to handle yourself in previous versions of eclipse - and let you select them from a nice GUI.

Friday, January 8, 2010

3D cinema explained

Avatar 3D is amazing.

Do you wonder how the 3D works? I did. Turns out it is distributed in several ways. Most cinemas use Dolby 3D or RealD. I got to see Avatar 3D in RealD, and it was nice!

The most important requirement to achieve actual depth perception is to get a different (distinct) picture to the each of the left and right eye, thus achieving stereopsis. This can be achieved in a variety of ways, but in cinema, the most obvious way (if not sole way, at this point in time), is to have the spectators wear glasses that separate two pictures that are displayed on top of each other on the same screen.

One way is to use colored glasses. The really old-skool is to use red/green or red/blue. This ends up loosing out on the blue or green component. Thus, a newer way is thus to use red / green-blue (cyan). These separate color inputs is somewhat fused in the brain, thus giving "full colors", but the effect is rather annoying, and you can literally feel the tearing in your brain as it tries to restore the 3D information from the spatially slightly differing images (but which have massively different color contents) and at the same time fuse the colors, into one single full color 3D image.

Dolby 3D uses an extended version of this colored glasses idea: it lets both eyes see red, green and blue - only different variations of the colors. The glasses is thus very specific in the bands they let through: One band of red, one band of green and one band of blue for the left eye, and another band of red/green/blue for the right - without having an overlap. According to the WP article, "Difference in color perception for the left and right eye are corrected within the glasses through additional filters" - this just needs to correct the relative strength of the r/g/b on one eye, and average on the other, since the eye doesn't distinguish between any two versions of red as long as they don't affect the other colors. To avoid using two projectors, a special color wheel that alternates between these two sets of red/green/blues is installed, so that each eye sees alternatively a black frame, or the image dedicated to it. A benefit of this system is that the screen can be any white surface, as it does not depend on any polarization effect of the light. Also, head-tilting does not pose a problem.

RealD, on the other hand, use polarized light. This way of separation needs a literal silver screen - yes, it is not just a figure of speech, there was actual silver involved. And now, with the needs of polarization-based 3D cinema, there is again. The obvious idea here is to use vertical polarization on one eye and horizontal on the other. This idea have been employed many times throughout the history of cinema. However, this has a problem when the viewer tilts his head: Since both glasses now are out of alignment, both eyes will see both images, effectively killing the 3D effect as if not wearing the glasses at all. RealD have fixed this, and this initially baffled me when checking out the glasses - by holding two sets of glasses over each other, one tilted 90 degrees, one could still see through them (try that with two sets of polarizing sunglasses!). They use circular polarization. Again, to not have to install two projectors, a device in front of the projector switches between the two polarizations (apparently not using a rotating wheel, even though this should be possible since the polarization is circular instead of horizontal/vertical, in which case a rotating device would not be practical).

Here's a "How stuff works" article on 3D glasses.

Here's a good forum post on this exact subject. He points out that 3D is not new in any way - "In the 20's, sound was a gimick. In the 30's, color was a gimick, in the 50's/60's 3D and widescreens where both gimicks. Widescreens caught on, 3D didn't."

And here's an article that compares the three techniques now in existence - IMAX 3D, RealD and Dolby 3D - for the same movie, Beowulf. He goes into a bit of technical detail. (IMAX 3D uses vertical/horizontal polarization, but with two distinct projectors, currently employing actual film.)

As an absurd aside, I happened across this article about some kind of shrimp, the Mantis shrimp, that "sees" circularly polarized light. Other animals can distinguish between different linearly polarized light, which for example is nice when you have to see through a water surface (the water surface suddenly vanishes - try this with a SLR camera with a polarizing filter installed, or with your polarizing sunglasses, tilting your head to one side or the other, when standing ashore trying to see the bottom of the sea). However, these guys can distinguish any polarization from circular, through elliptic, to linear, in any phase. What benefit would this have for the shrimp? As a researcher pointed out: "Some of the animals that the [mantis shrimp] like to eat are transparent, and quite hard to see in sea-water – except that they're packed full of polarizing sugars – which makes them light up like Christmas trees as far as these shrimp are concerned". The Mantis shrimp seems like an awesome killer - they have hyperspectral vision (vision that stretches out to ultraviolet and infrared), and can as mentioned distinguish any polarization, and have, depending on species, either built-in spears or clubs, which they can employ with an acceleration of 10,400 g and which acquire speeds of 23 m/s, about the acceleration of a .22 calibre bullet. This is so fast that cavitation bubbles form, which give a second shock - so if the guy misses you with its club, you might die from the shockwave from the collapsing cavitation bubbles! Nice that I don't have to be on the lookout for such dudes every day.

Tuesday, December 22, 2009

AVG Antivirus is a pest!

AVG Antivirus is itself a pest in several regards.

Coming home to mom and dad for Christmas, I find that their Firefox is eating at least 50% CPU (of this admittedly slow machine) when showing a blank page! So an already slow machine is now really dead slow.

Using Process Explorer (awesome tool, link is to Microsoft) on the process, I find that some thread with start address in MSVCR80.dll is eating all the CPU, while the process also seems to constantly spawn dozens of new threads per second with this dll as startpoint (they die at the same rate, so there is no buildup). Googling around I find a tip: Disabling the plugin AVG Safe Search immediately zeroes the CPU load (and also the thread spawning - there is no threads showing MSVCR80.dll anymore. The dll is apparently nothing more than MS C Runtime Library - I have no idea why it ends up like this, maybe it is just the thread entry point?).

AVG installs two plugins, one "AVG Safe Search", the one that kills your machine, and one "AVG Security Toolbar", which gives NOTHING besides providing a Yahoo search box, obviously for hope of revenue from sending searches Yahoo's way, and eating lots of screen estate. Amazingly, I find that AVG also changes the Search box of Firefox to Yahoo, and that as long as AVG Toolbar is enabled, it is impossible to change the Search box away from Yahoo, not even over to e.g. Wikipedia!! And I now also found that AVG has also hijacked the default browser search engine (when you type something into the location bar) to Yahoo - an option that you apparently need to go to about:config to change back. Also, it is not possible to remove the AVG Firefox plugins by means of Firefox's own plugin manager - but you can disable it.

In this regard, AVG is nothing more than a seriously nasty "browser toolbar".

Besides this buggy code and annoying toolbar, AVG really takes a boatload of time to run through less than half a TB of disk (even though none of the files changed), daily, and pretty much pegs the CPU at full throttle, and also "thrashes" all the memory of all other running processes onto disk, since it races through files and probably hence churns the OS's cache. The end result is that my own machine is completely unusable when "AVG is running scans" - so if I sit at it when it starts, I have to kill it, and if I come back to it when it still is running, I have to kill it. To top it, sometimes the tray icon hangs, so it is not possible to kill the scanning by normal means either (although you can just kill it from some process explorer still).

And finally, there is the concept of AVG Free. That is all nice and well and good, but the point here is that it is only free for about a year. After that, you have to update. You can update to a new installation of AVG Free, but they hide this fact very good. The point is obviously that they hope and assume that people like me, "tech savvy nerds", install this onto computers of their friends and relatives - it is free and all!! However, after about a year, these poor friends and relatives will CONSTANTLY be bugged about their machine being at risk blah blah, and redirected to an AVG update page. This page will pretty much force them to pay (unless they themselves are savvy enough to understand that beneath one of the extremely small graphics on that page they might find a new installation of AVG Free). So, yes, my father is currently an AVG customer - which in itself isn't bad, but he has been lured into becoming one - and I was the unwitting sales agent for AVG.

This is a technique I find bordering on adware.

I will never recommend AVG again.

Monday, March 23, 2009

Installing Debian on "Slug" NSLU2

Essence of this blogpost: Log in with a shell on the device to tail the syslog while the installer runs for much finer grained feedback during install. This can be OK, since it often takes "half a year" to format the disk and install the device, and one start to wonder whether it is hung.

How cool: You can install the actual Debian newest version 5.0 system onto a Slug!

This is a quick note on the installation of the Cisco Linksys NSLU2 device.

The debian install works like this: You "upgrade" the device "to the installer" using one of three methods, one of which is to simply use the web console of the device and direct that to the "di-nslu2.bin" file you downloaded.

The device then reboots, and after some minutes emits three beeps. You now log in to it using ssh (for Windowers, this means putty). The standard ip of any new Slug is 192.168.1.77, but if you enabled DHCP using the web console before you "updated the firmware", it will still use DHCP after booting, normally getting the address it got the first time (DHCP clients always asks for the same IP they got the first time from a DHCP server), or being given the static configured DHCP IP address you've configured on the DHCP server (which IMHO is the right thing to do).

The username/password is installer/install. You end up on the "Network console for the Debian installer". Here you can select either "install it damnit" or "install (expert mode)", or "start shell".

You select one of the install options, and you're on your way.

The point of this blogpost is that the installer takes some time (like in the 4 hour range!). And just the formatting of a 1TB disk takes at least a full hour, where at least 95% of the time the progress bar shows "33% finished"! This makes one wonder what is happening.

What is cool, is that one can log in once more to the installer. This time, select "Start shell". Now tail the syslog:

cd /var/log
tail -f syslog


.. or, when the device is formatting your disk, which takes ages, instead tail "partman". Ctrl-C to stop. Since the Slug has very little memory, you should not do much creative stuff with the shell you've got there during the install, or else the installer itself might be terminated by Linux' OutOfMemoryKiller.

Tuesday, March 17, 2009

Jonathan Schwartz censors blog comments!!

Wasn't that a "WOW! Are you kidding??"-title? I threw in a comment on Jonathan Schwartz's blogpost, but I guess it was way to radical to slip through the censorship.

Update 2009-03-18, regarding the comment "Hope it pans out": IBM might buy Sun, so maybe this isn't exactly panning out at all! I find this sad - I've always appreciated this Standford University Network "spinoff" and the stuff they have produced. It will be a pretty hard smack in the face for the Open Source business concept if Sun now completely fails with Schwartz' large idea.

" Interesting blogposts; frank and open. I really enjoy them. The logic even seems sound, I hope it pans out!

However, seeing that you changed your ticker to JAVA and all, it clearly seems you think Java actually has a bit of value. I would thus love for you to comment on and explain why JavaFX is a better idea than actually making java on the desktop better and potentially also keep up the idea behind lwuit for those "other screens of my life" (or however that jingle goes) which actually seemed to have picked up a small momentum, or at least interest.

I bet Microsoft and Apple are having a great time for each blunder you do with java on the desktop: You could by this time "owned" much of the development on the desktop and we could all thereby had proper crossplatform software, had you focused on making the java platform "agile" in a much larger way: Footprint reduction and heavy modularization, startup time closer to native and flash, automatic updates, cross platform, media, Swing improvements and new components, applet handling and the list goes on. These things should have been driven a long time ago; they have been glaring and obvious problems all the time. Some of these things you JUST started doing, WAY late but still, with java 6 u10 and the direction this was taking, and then, absurdly, you totally switch direction into JavaFX, diverting focus and resources (and coincidentally it seems several of your best employees in these areas quit), and not least actually fragmenting java GUI'ing into a completely new language with components not easily used in the "old java", to the exceptional annoyance of, as I've understood it, /very/ much of your following. It thus time and time again seems like you pretty much ignore feedback from the community, instead going into totally irrelevant directions - loosing momentum and, importantly, lots of time.

The fundament and the idea behind Java is so good, and the problems I listed aren't that huge - they just need focus, as update 10 showed - so it is just sad to see how you again and again seem to miss the /relevant/ boats! I sadly believe JavaFX will be an extremely tough hill to climb - why on earth would one use anything else than flash? Even Sun themselves use flash for all their presentations - even on these very blogposts. If anything, you might have helped drive the opening of flash and that open screen project, but I guess that wasn't the real aim. And in the meantime, everything else is delayed. Other actors are now taking over the leadership of Java, potentially (and already) fragmenting the place. And as an end comment; What's that absurd situation with Apache Harmony supposed to mean?! - that in no way moves Java forward. "

Saturday, March 14, 2009

JCP's EC transparency

This is hilarious:
(From EC September 2008 Meeting Minutes)

Transparency in EC meetings

After a brief discussion of the need to make EC meetings more accessible to JCP members and to the general public, Roberto Chinnici proposed the following motion:

"Minutes of EC meetings (including presentation materials) shall by default be public rather than EC-confidential. The EC will decide by a simple majority vote of those present whether or not to go into private session during which no votes may take place, but for which EC-private rather than public minutes will be published."

Vicki Shipkowitz seconded the motion, which was passed unanimously.

The ECs also agreed that Meeting Minutes will be accessible to all (not just to JCP members). The PMO will remove the password-protection from all previously-published Meeting Summaries, and will remove any "JCP EC Confidential" notices from these summaries. Previously-published Meeting Minutes will continue to be accessible only to EC members. These changes will be effective immediately.

Private Session

The ECs then went into private session for a discussion on the negotiations between Apache and Sun.


HAHAHA!

Tuesday, March 10, 2009

Friends for Java v2

Here's a new attempt at implementing a secure version of something akin to the friend class concept in java, since the previous attempt was rather flawed in view of being used in a "neat'n'clean API" scenario.

This time I'm using a little introspection to try to catch evil code. Hopefully this check can't be circumvented, at least if a SecurityManager is in place, and also hopefully won't be stopped by any SecurityManager (it doesn't immediately seems so: one can use getClass() at any time, right? Can one also use the 1.5 getEnclosingClass() without being stopped?)

(If you use Eclipse, you can mark the entire code below (all classes in one go), Ctrl+C, then activate the project node of a project and hit Ctrl+V. You will probably have to organize imports afterwards).
package com.example;

import api.WantedCode;

public class NeedingCode {
// Driver
public static void main(String[] args) {
// .. we have an instance of this class
WantedCode code = new WantedCode();

// And access its package private method through a "friend-proxy"
FriendProxy.getProxy().invokeWantedMethodOf(code);
}
}

// ------------

package com.example;

import api.WantedCode;

public abstract class FriendProxy {

// :: Proxy interface method

public abstract void invokeWantedMethodOf(WantedCode instance);

// :: Friend infrastructure

private static FriendProxy _proxy;

public static void setProxy(FriendProxy proxy) {
// Verify that the origin of this proxy matches our expectations.
if (proxy.getClass().getEnclosingClass() != WantedCode.class) {
throw new IllegalAccessError("Only my friend can invoke this method.");
}
_proxy = proxy;
}

// :: Package-private accessor to get to friend's package private method.

static FriendProxy getProxy() {
return _proxy;
}
}

// ------------

package api;

import com.example.FriendProxy;

public class WantedCode {

void wantedMethod() {
System.out.println("Hi from the package-private method in the API!");
}

static {
FriendProxy.setProxy(new GivingAccessToNeedingCode());
}

private static class GivingAccessToNeedingCode extends FriendProxy {
@Override
public void invokeWantedMethodOf(WantedCode instance) {
instance.wantedMethod();
}
}
}

// ------------

package com.example;

import api.WantedCode;

/**
* Test whether the evilness is caught.
*/
public class EvilCode {
// Driver
public static void main(String[] args) {
WantedCode code = new WantedCode();
FriendProxy.setProxy(new FriendProxy() {
@Override
public void invokeWantedMethodOf(WantedCode instance) {
System.err.println("I'm evil!");
}
});
FriendProxy.getProxy().invokeWantedMethodOf(code);
}
}

Monday, March 9, 2009

Friends for java

I read a blog entry of Roman Kennke. He's talking about implementing something akin to the friend class concept in java. The problem goes like this:
package api;

public class WantedCode {
void wantedMethod() {
System.out.println("Wanted method in com.example");
}
}

// --- in another package ---

package com.example

import api.WantedCode;

public class NeedingCode {
void needingMethod() {
// .. we have an instance of this class
WantedCode instance = new WantedCode();
// And need to access some package private method in that class
instance.wantedMethod() // <- error, since it is package private
}
}
The solution is to make an interface within NeedingCode's package, which declares a method that will invoke the package private method in WantedCode, taking as argument the instance of WantedCode on which the method shall be invoked. This interface will be implemented somewhere in WantedCode's package, and then statically set somewhere in NeedingCode's package. The corresponding getter is package private, and one has thereby established a "friend link" for this method which only can be used by NeedingCode's package.

I had some problems following the post since the code snippets was so fragmented, so here I've dumped the full code, and also hopefully answered my own question put forward in the post: Anyone could set the proxy instance, thereby redirecting NeedingCode's invocations to some evil code. The idea is to use a two-way "handshake" to make sure that only the friend sets the proxy, by use of a "secret" (a private Object instance).

Update: Roman Kennke pointed out the obvious: This pretty much ruins the original intent, as one then have ended up with a public facing "magic method" that is shown in the API (and one could then basically just have let the package-private method be public instead). On the other hand, this method can only be used for the sole purpose of establishing the specific friend aspect, and only to the selected other package - none other can make any use of it. Furthermore, if one can accept one such public facing static magic method in the API-package, one could use this method to bridge this package (and any package-private method in that package) to any other (internal) packages.

I've made a second attempt! That one uses introspection to verify the origin of the supplied proxy instance.

(If you use Eclipse, you can mark the entire code below (all classes in one go), Ctrl+C, then activate the project node of a project and hit Ctrl+V. You will probably have to organize imports afterwards).
package com.example;

import api.WantedCode;

public class NeedingCode {
// Driver
public static void main(String[] args) {
// .. we have an instance of this class
WantedCode code = new WantedCode();

// And access its package private method through a "friend-proxy"
FriendProxy.getProxy().invokeWantedMethodOf(code);
}
}

// ------------

package com.example;

import api.WantedCode;

public abstract class FriendProxy {

// :: The proxy interface aspect

public abstract void invokeWantedMethodOf(WantedCode instance);

// :: "Handshake" that establishes friendship

private final static Object _secret = new Object();

static {
WantedCode.makeFriends(_secret);
}

private static FriendProxy _proxy;

public static void setProxy(FriendProxy proxy, Object secret) {
if (secret != _secret) {
throw new IllegalAccessError("Cannot set proxy without correct secret.");
}
_proxy = proxy;
}

// :: Package-private accessor to get to friend's package private method.

static FriendProxy getProxy() {
return _proxy;
}
}

// ------------

package api;

import com.example.FriendProxy;

public class WantedCode {

void wantedMethod() {
System.out.println("Hi from the package-private method in the API!");
}

// :: 2nd part of "handshake" that establishes friendship

public static void makeFriends(Object secret) {
FriendProxy.setProxy(new GivingAccessToNeedingCode(), secret);
}

private static class GivingAccessToNeedingCode extends FriendProxy {
@Override
public void invokeWantedMethodOf(WantedCode instance) {
instance.wantedMethod();
}
}
}

Tuesday, March 3, 2009

AWT Swing Event Pumping and Targeting

I'm currently learning Swing, and thus AWT. In the post "Debug Listeners on AWT/Swing components", I wrote a little piece of code that can help with understanding what events are fired on a component.

This is a further dive into the lower level elements of the event system of AWT/Swing. I wrote this as research notes for myself, but have then tried to edit it into something more readable, hoping that a wider audience than me alone can get any value from it.

Intended audience: Somewhat experienced developers of the type that likes to know how things really work, and who subscribe to Joel's law of Leaky Abstractions. If you're a AWT/Swing guru that for example know how the current version of AWT handles the AWT 1.0 event model and where exactly in the event dispatch and processing chain the AWTEventListeners are invoked, you probably don't need this. If you've never written some hello world for Swing, it might be too early - but you could skim it nevertheless, to know if it will be interesting for you later.

In the following text, which is a long article more than a blogpost, I ended up covering quite a bit. The main elements are:
  1. Basics of AWTEvents, low-level events and semantic (high-level) events.
  2. Event pumping, Operating System to Java interaction and the transfer of events.
  3. Event dispatching, Event Dispatch Thread ("EDT")
  4. Event processing.
  5. InputEvent (mouse and keyboard) retargeting, Focus subsystem
  6. Some misc topics: Coalescing, AWTEventListeners, Key Bindings, and how semantic events are produced and dispatched (Spring's MVC logic).
Please be advised that there might be places where I haven't understood things correctly! The text is neither supposed to be a complete discussion of all the intricacies of Swing/AWT. However, the text might hopefully, for some level of developers, shed light on some elements of "how it really works". The text is supposed to progress in a somewhat orderly fashion - if it doesn't, just read it a couple of times more! If you have any corrections, ideas for making it easier to understand this stuff, or otherwise have any suggestions, please comment - or send an email to Endre@Stolsvik.com.

The text contains a bunch of stacktraces. These are taken from runs on Java 1.6.0_12. However, these parts of the code don't change radically from version to version, and the overall discussion and even the stacktraces (less the exact line numbers) will probably still be relevant in some years from now.

Basics

Different types of events: The event system comprises two distinct types of events: Low-level events, and Semantic events. The low-level events all extend ComponentEvent, while the semantic events are all others, for example ActionEvent, AdjustmentEvent, ItemEvent, TextEvent and others. The low-level events are direct events concerning the GUI system and represent window-system occurrences or low-level input, like a window being moved or minimized, a key being pressed on the keyboard, the mouse being moved, a mousebutton being clicked. The semantic events, on the other side, pretty much all describe changes of state on Components, like a button being clicked, a menu item being choosen, text being entered into a field. Semantic events are typically constructed and dispatched by the components themselves based on low-level events.

Here's a rather old (Feb 1997) article from Sun: "Java AWT: Delegation Event Model", concerning the change from Java 1.0 event model to the Java 1.1 delegation event model. This model is still the one being employed, and the article is a rather easy read - just decide if the historic parts are worth paying attention to or not. Here's a link to the Swing UI tutorial about these two kinds of events: "Concepts: Low-Level Events and Semantic Events". Here's a link to the event package JavaDoc: Package java.awt.event.

This article will primarily concern the low-level events.

All AWT/Swing events are of supertype AWTEvent: The events fired on the different listeners are extensions of AWTEvent, for example MouseEvent, and the specific subtype of mouse event is communicated by which of the different methods on the MouseListener (or MouseMotionListener or MouseWheelListener) is invoked, for example MouseListener.mouseClicked(MouseEvent e). AWTEvent also has a method getID() that describes which type (e.g. "mouse") and subtype (e.g. "mouse moved") the event is (MouseEvent ids starts at 500, and MOUSE_MOVED is 503). AWTEvent is a subclass of Event, which has a getSource() method. The source always refers to the component "in question": Low-level events originate from the windowing system, and the source is the component that got or will get the Event, and it can thus quite cleanly be thought of as the target. High-level events are typically created by the components themselves, and are thus originating from that component, making "source" correcter. When processing events, the EventDispatchThread, which is discussed later, directly invokes getSource() on the event, and then invokes source.dispatchEvent(event). The specific event classes add event type specific stuff, for example MouseEvent which amongst others has methods getLocationOnScreen() and getButton().

Here's the Sun tutorials index for EventListeners: "Lesson: Writing Event Listeners", which has lots of good information that hopefully will make even more sense after reading this article. In particular the listing in "Listeners Supported by Swing Components" is good for an overview or index over what event listeners are supported by which components. Note how it divides the list into low-level events ("Listeners that All Swing Components Support" - the AWT events) and higher-level semantic event listeners ("Other Listeners that Swing Components Support"). From this page you can get a pdf of Chapter 9 of the book "Mastering the JFC: AWT, Volume 1", which is a 2001 book but which still is relevant.

From the Operating System to the MouseListener

Native Event Loop: The Toolkit implementation and its corresponding event pumping thread is the native connection to the underlying operating system, and is thus specific for each operating system. Inside the Toolkit thread event loop, extension classes of AWTEvent (e.g. MouseEvent) are instantiated and posted to the EventQueue. The source is set to the appropriate AWT Window: For example, for KeyEvents, this is the active window, while for MouseEvents, it is the window below the mouse. The AWTEvent is "posted" to the java side by some roundabout ways which effectively ends up in an invocation of EventQueue.postEvent(AWTEvent). The specific Toolkit instance along with the toolkit thread forms the native side of the GUI event pumping.

The source code for the native Toolkit implementations aren't available in the standard SDK distribution. Here's a link to WToolkit, the Windows Toolkit implementation. Notice the bunch of native methods. Notice the native method eventLoop() at line 303 and the call to this from the Runnable of the "AWT-Windows" thread at line 289.

Here's a stacktrace of a posting of a MouseEvent.MOUSE_MOVED event from the native Toolkit thread, named "AWT-Windows". Notice how it doesn't post it directly onto the EventQueue, but rather onto a "intermediate storage point" called PostEventQueue:

 Daemon Thread [AWT-Windows]
  sun.awt.PostEventQueue.postEvent(java.awt.AWTEvent) line: 2083
  sun.awt.SunToolkit.postEvent(sun.awt.AppContext, java.awt.AWTEvent) line: 591
  sun.awt.windows.WFramePeer(sun.awt.windows.WComponentPeer).postEvent(java.awt.AWTEvent) line: 722
  sun.awt.windows.WToolkit.eventLoop() line: not available [native method] [local variables unavailable]
  sun.awt.windows.WToolkit.run() line: 291 java.lang.Thread.run() line: 619

Java Event Loop: The EventDispatchThread ("EDT") is a thread that basically repeatedly invokes EventDispatchThread.pumpOneEventForFilters(int id) in a loop. This again invokes EventQueue.dispatchEvent(AWTEvent e). The EventQueue and EventDispatchThread together forms the java side of the event pumping. Upon dispatch, the EventQueue does checks what type of Event is being handled. ActiveEvents has a method dispatch() which is invoked directly. If the source of the AWTEvent is a Component, this component has its dispatchEvent(AWTEvent) invoked with the event as the argument.

The EventQueue gets events posted either from the native Toolkit thread or from the application code ("userland"). EventQueue.invokeLater(Runnable) and invokeAndWait(Runnable) posts an event to the EventQueue. Such events will be of type InvocationEvent (which is an ActiveEvent). It is also possible to directly post events onto the queue, as such: Toolkit.getDefaultToolkit().getSystemEventQueue().postEvent(event), but getting the EventQueue might be denied by the call to System.getSecurityManager().checkAwtEventQueueAccess(). Note that a lot of event firing, particularly for the higher-level events (the Semantic events), doesn't go through the EventQueue. For example, if a JButton is clicked, the generated ActionEvent is never posted to the EventQueue, but instead fired on the button's ActionListeners directly ("directly" is relative; The event is fired through or rather by Swing's Model-View-Controller paradigm, but still not through the EventQueue - more on this later).

Continuing on the MouseEvent.MOUSE_MOVED example started above: The EventQueue is notify'ed, leading to a flush of events from the PostEventQueue into the EventQueue. The thread here is the java side, the EventDispatchThread, named "AWT-EventQueue-0" (The zero at the end of the name is increased if this thread crashes, in which case a new one is created when the next event is posted by the Toolkit, and also increased for modal dialogs):

 Thread [AWT-EventQueue-0]
  java.awt.EventQueue.postEvent(java.awt.AWTEvent, int) line: 244
  java.awt.EventQueue.postEventPrivate(java.awt.AWTEvent) line: 202
  java.awt.EventQueue.postEvent(java.awt.AWTEvent) line: 175
  sun.awt.PostEventQueue.flush() line: 2072
  sun.awt.SunToolkit.flushPendingEvents() line: 626
  java.awt.EventQueue.getNextEvent() line: 465
  java.awt.EventDispatchThread.pumpOneEventForFilters(int) line: 236
  java.awt.EventDispatchThread.pumpEventsForFilter(int, java.awt.Conditional, java.awt.EventFilter) line: 184
  java.awt.EventDispatchThread.pumpEventsForHierarchy(int, java.awt.Conditional, java.awt.Component) line: 174
  java.awt.EventDispatchThread.pumpEvents(int, java.awt.Conditional) line: 169
  java.awt.EventDispatchThread.pumpEvents(java.awt.Conditional) line: 161
  java.awt.EventDispatchThread.run() line: 122

Two-thread event pump: Notice how there are thus two completely separate threads doing event pumping: One running on the native side, the Toolkit thread, pumping events from the native operating system "over to java" via posting to an intermediate PostEventQueue class, and then notifying the java side via the EventQueue. On the java side, we have the EventDispatchThread that waits on the EventQueue. When it gets a notify, it first flushes any new events from the native side into the EventQueue proper, and then pumps from the EventQueue.

Dispatching on the AWT and Swing Components: The main dispatching method on to the AWT/Swing component side is Component.dispatchEvent(AWTEvent e), which is final, but invokes Component.dispatchEventImpl(AWTEvent) which is package-private and is overridden by Container and furthermore by Window. Handled events in these overrides include ComponentEvent.COMPONENT_RESIZED for Windows, in which case it invalidates and validates before passing on to super, and resize and move events for Container, in which case it produces HierarchyEvent.ANCESTOR_[RESIZED|MOVED] for its children, after it has invoked super. However, Container also takes care of all MouseEvents, doing retargeting and redispatch, which we will come back to.

The Component.dispatchEventImpl(AWTEvent) is a rather long method that handles different aspects of different events, divided into steps. It takes care of a lot of special cases, both because of special requirements for some types of events, for the KeyboardFocusManager and lightweight component retargeting for events handled by focus, and for flexibility, but also because of lots of cruft in the event handling accumulated throughout the years (In particular this goes for the Java 1.0 AWT style of event handling (explained in the article "Java AWT: Delegation Event Model" referenced above) whereby one had to subclass a component to get events, which then were delivered by the Component's "self-invocation" of for example the overridden Component.mouseDown(Event e)). The Component.dispatchEventImpl(e) are nicely commented inline, clearly showing each stage of the dispatch. Unless some special case kicks in (of which there are many, in particular input event retargeting, described shortly), the method processEvent(e) is eventually invoked in step "6. Deliver event for normal processing", but only if Component.eventEnabled(AWTEvent e) returns true. This method returns true if either the event-type specific listener field is non-null, or if the eventMask, set by enableEvents(long), says so. The processing is described shortly.

InputEvents (re)targeting: The (currently) two types of InputEvents have distinct targeting logic, meaning which Component's Listeners shall be invoked. MouseEvents goes to the most specific ("deepest") child component residing under the mouse, while KeyEvents are sent to the focused window, routed using the KeyboardFocusManager to the focused component in this window.

MouseEvents targeting: Container overrides dispatchEventImpl(AWTEvent). Remember that Window is a Container. If the Container in question is a heavyweight component (i.e. it is not an instanceof LightweightPeer), this override takes care of finding the target for MouseEvents, attempting via an instance of the class LightweightDispatcher to forward/distribute MouseEvents to the most specific ("deepest") child, whether this is a heavyweight AWT Component, or a lightweight JComponent. These days this class has an incorrect name, since it also is responsible for retargeting MouseEvents to any heavyweight Components. (The class is package private, residing in Container.java).

To find the deepest component, Container.getMouseEventTarget(x, y ...) is invoked, which goes through each child component, finding the one that returns true for Component.contains(x, y). If the child is a Container, it recurses by invoking getMouseEventTarget(x - comp.x, y - comp.y ...) on the child (hence upholding the "illusion" that 0,0 is the local top left corner for all component). Since JComponents extends from Container, this method also checks itself after the recurse. The check mentioned is whether it wants any MouseEvents by testing whether the eventMask says so (mentioned above), or if any of the three types of mouse listeners are set (Mouse, MouseMotion, MouseWheel). The net effect is that the LightweightDispatcher thus finds the deepest Component which the mouse is currently over and which wants MouseEvents, retargets the event by making a new one, dispatches the new event instance directly, and consumes the original event instance. Or it finds that there are no such component, either because there physically aren't any components there (the window background is showing), or because the component don't want MouseEvents (for example a standard JLabel doesn't), in which case the LightweightDispatcher won't do anything (it won't consume the event), and the event will be dispatched as normal (on itself, the native Container, typically a Window).

LightweightDispatcher also tracks mouse enter and exits over lightweight components as the native system doesn't know about them and thus cannot make enter/exit events for them either. Also, if we're dragging, it hooks an AWTEventListener on the Toolkit (as mentioned below), so that it gets all further MouseEvents even though we drag across different boundaries of lightweight and heavyweight components.

If the original event is consumed, the LightweigthDispatcher dispatchEvent returns true, and the Container.dispatchEvent returns. If it doesn't consume the event (hence, it was no MouseEvent, or it was a MouseEvent targetted at the Container itself), it returns false, and the Container.dispatchEvents continues by invoking super.dispatchEventImpl(AWTEvent e), and hence normal dispatch ensues.

KeyEvent targeting / Focus Subsystem: As briefly mentioned, Component.dispatchEventImpl(e) also handles keyboard targeting: KeyboardFocusManager's static method retargetFocusEvent() is invoked, apparently to handle actual FOCUS_GAINED and FOCUS_LOST events. Then, the KeyboardFocusManager is statically asked for the current KeyboardFocusManager (I am not sure if it really is possible to entirely roll your own KFM, as the abstract class KFM specifically states several places that it depends on the code in DefaultKeyboardFocusManager), and dispatchEvent(e) is invoked on that. If this invocation returns true, the event was dispatched, and we are finished. It is possible to register KeyEventDispatcher instances on the KeyboardFocusManager, which can consume the event - the focused component is already established at that point, being set in the KeyEvent (getSource(), or rather getComponent() since it is a ComponentEvent - read towards the top, "All AWT/Swing events are of supertype AWTEvent"). Here's the Sun Java tutorial: "How to use the Focus Subsystem". For a in-depth description of the focus subsystem, read the article The AWT Focus Subsystem, referenced from the Component class and in particular its processKeyEvent method. This subsystem is somewhat complex, including logic about multiple focus cycle roots and how the different components are notified about focus gained and lost events. An interesting note is that if you end up in a more "inner" focus cycle root, there are by default no keypresses that can lift you out to the outer root again.

Note that for JComponents, there is another way to hook Actions to specific KeyStrokes, namely the InputMap/ActionMap system, which is described below.

Event processing on Component: As mentioned above, Component.processEvent(AWTEvent e) is eventually invoked. It has a bunch of "downstream" methods: process[Component|Focus|Key|Mouse[Motion|Wheel]|Input|Hierarchy[Bounds]]Event. All these methods can be overridden in subclasses, thus providing interesting (and non-trivial) hook-points. If you do override them, remember enableEvents(bitfield), described shortly. It is these downstream methods, for example processMouseEvent, that eventually invokes the listeners, for example MouseListener.

For any type of event, there can be several EventListeners on a Component. In the Component class, there is a sole field for each type, e.g. keyListener. Nullness of this field is used as indicator: If it is null, there is no listeners for that event type. If it is non-null, there is at least one. If there is one listener, it will be the actual listner. If there are several listener, the field will be an AWTEventMulticaster instance, which is a somewhat clever way of simply daisy-chaining a set of listeners of the same type into a string of listeners whose event fire methods all will be invoked when the AWTEventMulticaster's corresponding fire-method is invoked. AWTEventMulticaster obviously implements all types of AWT EventListeners. The order in which listeners of a specific type is invoked is indeterminate, thus if important, you must handle this in some way yourself.

Events are not delivered to a Component unless there are at least one listener for this event, or if enableEvents(long eventsToEnable) is invoked (in which case processEvent(e) is invoked even though there are no listeners for that type). The long is a bit-mask, composed by ORing together the constants in AWTEvent. The process[Type]Event does the null-check on the corresponding typeListener field, returning immediately if null. If non-null, it invokes the correct method on the TypeListener based on a switch on event.getID(). All process[*]Event methods are protected and non-final, hence overridable for subclasses.

MouseEvent.mouseClicked: Here is a stacktrace for invocation of mouseClicked(e) on a MouseListener installed on a JLabel when I clicked the mouse. The stacktrace begins in the EventQueue where an event is pumped. It is dispatched on the Window instance where it goes to the Window and Container specific overrides of dispatchEventImpl. It is a MouseEvent, so it goes through the LightweightDispatcher where this MouseEvent is retargeted (a new MouseEvent is created, the old is consumed). The LightweigthDispatcher directly redispatches the new MouseEvent on the JLabel instance where it is processed. Finally, it ends up in the ExampleMouseListener as an invocation on its mouseClicked(e) method. (In this example, the JComponent also intervenes to handle "Autoscroll". This is a feature that enables moving of a scrollpane by dragging the content: If you setAutoscrolls(true), synthetic MouseDragged events will be made if you click inside the component and then drag the mouse, even though you drag it outside the component. The override of the processMouseEvent is to stop this feature when you release the mouse button.). Here's Sun's tutorial on MouseListeners: "How to write a Mouse Listener". The thread running is, as always for Java GUI stuff, the EventDispatchThread.

 Thread [AWT-EventQueue-0]
  com.example.ExampleMouseListener.mouseClicked(...)
  java.awt.Component.processMouseEvent(Component.java:6219)
  javax.swing.JComponent.processMouseEvent(JComponent.java:3265)
  java.awt.Component.processEvent(Component.java:5981)
  java.awt.Container.processEvent(Container.java:2041)
  java.awt.Component.dispatchEventImpl(Component.java:4583)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ This is a redispatch of a new MouseEvent, now on the JLabel, while the old event will be consumed.
  java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4556)
     ^^ It has now found the target Component: The JLabel
  java.awt.LightweightDispatcher.processMouseEvent(Container.java:4229)
  java.awt.LightweightDispatcher.dispatchEvent(Container.java:4150)
     ^^ The retargeting of the MouseEvent begins, by the LightweightDispatcher held by the Window instance.
  java.awt.Container.dispatchEventImpl(Container.java:2085)
  java.awt.Window.dispatchEventImpl(Window.java:2475)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ The initial dispatch, on the Window that holds the JLabel.
  java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
  java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)    <-- "EDT pump" references!
  java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
  java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
  java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
  java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
  java.awt.EventDispatchThread.run(EventDispatchThread.java:122)

KeyEvent.keyTyped: To contrast, here's a stack trace for invocation of keyTyped(e) on a KeyListener installed on a JButton. I made the button focused (by clicking on it, or by tabbing to it), then hit a key. The event is processed the same up to the Container.dispatchEventImpl, where the event is not processed, and instead goes further to Component.dispatchEventImpl. Here, the KeyboardFocusManager kicks in since it is a KeyEvent, and does its magic focusing logic and where the KeyEvent is retargeted (a new KeyEvent is created, the old is consumed). The KeyboardFocusManager directly redispatches the new KeyEvent on the JButton, where it is processed. Finally, it ends up in the ExampleKeyListener as an invocation on its keyTyped(e) method. Here's Sun's tutorial on KeyListeners: "How to write a Key Listener".

 Thread [AWT-EventQueue-0]
  com.example.ExampleKeyListener.keyTyped(...)
  java.awt.Component.processKeyEvent(Component.java:6171)
  javax.swing.JComponent.processKeyEvent(JComponent.java:2799)
  java.awt.Component.processEvent(Component.java:5993)
  java.awt.Container.processEvent(Container.java:2041)
  java.awt.Component.dispatchEventImpl(Component.java:4583)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ This is a redispatch of a new KeyEvent, now on the JButton, while the old will be consumed.
  java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1848)
     ^^ It has now found the target Component: The JButton
  java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:704)
  java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:969)
  java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:841)
  java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:668)
     ^^ The retargeting of the KeyEvent begins, by the global KeyboardFocusManager
  java.awt.Component.dispatchEventImpl(Component.java:4455)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Window.dispatchEventImpl(Window.java:2475)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ The initial dispatch, on the Window that holds the JButton.
  java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
     ["EDT pump"]


Misc topics

Coalescing: This is a mechanism whereby a bunch of e.g. mouse-move events are compressed into one. This is done by checking whether the event currently-being-posted-to-the-Event-Queue could be collapsed into an already-posted-but-not-yet-processed event. Earlier, this was a general system whereby the source ("target") component would possibly go through the entire eventqueue looking for same-type events, checking whether it could collapse them, but this has been deemed too slow. The EventQueue.coalesceOtherEvent method that takes care of this legacy has the following comment: "Should avoid of calling this method by any means as it's working time is dependant on EQ length. In the wors case this method alone can slow down the entire application 10 times by stalling the Event processing. Only here by backward compatibility reasons." The present system uses the field Component.eventCache, where five specific event types can be handled: PaintEvent.PAINT, PaintEvent.UPDATE, MouseEvent.MOUSE_MOVED and MouseEvent.MOUSE_DRAGGED, in addition to sun.awt.PeerEvent (which I don't know what is) which apparently handles coalescing itself (it also handles dispatching itself - it is an ActiveEvent, specifically an extension of InvocationEvent). The coalescing is as dumb (and fast) as possible: mouse events are coalesced by keeping the latter event, while the paint events are coelesced by checking whether one or the other "update rect" is contained in the opposite, in which case it returns the containing, or else it doesn't coalesce.

AWTEventListener, "Event spying": On the Toolkit, one may install a "global listener" to spy on all low-level events that are dispatched from the EventQueue onto some Component: Toolkit.getDefaultToolkit().addAWTEventListener(listener, long eventMask), where the eventMask is the same bit-mask described Component.enableEvents(...) above. Adding an AWTEventListener might be denined by the call to System.getSecurityManager().checkPermission(SecurityConstants.ALL_AWT_EVENTS_PERMISSION).

This is a stacktrace when such an AWTEventListener is invoked, also on the EventDispatchThread, illustrating both the EventQueue and the run through the dispatching on the Window, Container, Component. The invocation of AWTEventListeners are done early in the Component.dispatchEventImpl, at step "2. Allow the Toolkit to pass this to AWTEventListeners", which makes it possible for a AWTEventListener to consume the event early and thereby stop the dispatch process from going further.

 Thread [AWT-EventQueue-0]
  com.example.ExampleAWTEventListener.eventDispatched(...)
  java.awt.Toolkit$SelectiveAWTEventListener.eventDispatched(Toolkit.java:2353)
  java.awt.Toolkit$ToolkitEventMulticaster.eventDispatched(Toolkit.java:2244)
  java.awt.Toolkit.notifyAWTEventListeners(Toolkit.java:2203)
     ^^ The Toolkit is invoked to pass the event through the AWTEventListeners.
  java.awt.Component.dispatchEventImpl(Component.java:4481)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Window.dispatchEventImpl(Window.java:2475)
  java.awt.Component.dispatchEvent(Component.java:4413)
  java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
     ["EDT pump"]

Key Bindings / Keyboard handling by InputMap and ActionMap: Swing components, JComponents, have one more way of handling KeyEvents: The InputMap and ActionMap. The InputMap defines a Map between KeyStrokes and "actionMapKeys" which (typically) are Strings that describes what this key does, while the ActionMap defines a Map between those actionMapKeys and some Action. Note that the actionMapKey arguement type is Object, but it is custom to use Strings so that these Maps become "self documenting".

Read the JavaDoc of the static methods on KeyStroke carefully; They are a little devious! The whole key handling system reeks of legacy. The problem spots revolve around the fact that for KEY_TYPED events, the KeyEvent instance returns the typed character in the method getKeyChar(), while for KEY_PRESSED and KEY_RELEASED events, one needs to use the getKeyCode() (which returns a KeyEvent.VK_[key] constant). This also makes for interesting times when using the getKeyStroke methods - be sure to read the JavaDocs. In particular, both the methods getKeyStroke(int keyCode, int modifiers) and getKeyStroke(int keyCode, int modifiers, boolean onKeyRelease) return strokes for KEY_PRESSED and KEY_RELEASED (the latter if the boolean is true), meaning that there are no way to get a stroke for KEY_TYPED when using the key codes.

This binding system is the functionality used by mnemonics (picking a menu item from a selected menu by a key, typically this letter is underlined) and accelerators (picking some menu item without navigating the menu, e.g. "Ctrl-F" for File->Open). There is also a trick here for the UI (the Look and Feel): The UI have a separate parent Map that takes care of any Look'n'Feel specific key bindings, and makes it possible to change the look and feel without loosing the application specific bindings.

Here's the JavaDoc from KeyboardManager, a package private helper class for the key bindings system, which I found nicely summing up the keybindings system:

" The KeyboardManager class is used to help dispatch keyboard actions for the WHEN_IN_FOCUSED_WINDOW style actions. Actions with other conditions are handled directly in JComponent.

Here's a description of the symantics of how keyboard dispatching should work atleast as I understand it.


KeyEvents are dispatched to the focused component. The focus manager gets first crack at processing this event. If the focus manager doesn't want it, then the JComponent calls super.processKeyEvent() this allows listeners a chance to process the event.


If none of the listeners "consumes" the event then the keybindings get a shot. This is where things start to get interesting. First, KeyStokes defined with the WHEN_FOCUSED condition get a chance. If none of these want the event, then the component walks though it's parents looked for actions of type WHEN_ANCESTOR_OF_FOCUSED_COMPONENT.


If no one has taken it yet, then it winds up here. We then look for components registered for WHEN_IN_FOCUSED_WINDOW events and fire to them. Note that if none of those are found then we pass the event to the menubars and let them have a crack at it. They're handled differently.


Lastly, we check if we're looking at an internal frame. If we are and no one wanted the event then we move up to the InternalFrame's creator and see if anyone wants the event (and so on and so on).
"

Since this article has come to include a bunch of stack traces to illustrate things, I'll chuck in a couple more to show where the keybindings kick in the event processing. On a window with two buttons, I've bound F2 to JButton A "WHEN_FOCUSED", F3 to the containing JPanel "WHEN_ANCESTOR_OF_FOCUSED_COMPONENT", and finally F4 to the same JButton A "WHEN_IN_FOCUSED_WINDOW". Hitting key F2, F3 and F4 when the button A is focused "hits" on all keys, while if JButton B is focused, F2 doesn't fire, but F3 and F4 still does; F2 doesn't fire since it is not focused, F3 fires since it is bound to the ancestor JPanel of JButton B, while F4 fires since the window that JButton B resides in is focused.

Here's the stacktrace for F2, which is bound to the JButton A "WHEN_FOCUSED", and fires when JButton A has focus:

 Thread [AWT-EventQueue-0]
  com.example.ExampleAction.actionPerformed(...)
  javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1636)
  javax.swing.JComponent.processKeyBinding(JComponent.java:2849)
  javax.swing.JComponent.processKeyBindings(JComponent.java:2884)
      ^^ Checking own bindings
  javax.swing.JComponent.processKeyEvent(JComponent.java:2812)    <-- .. next trace starts here!
  java.awt.Component.processEvent(Component.java:5993)
  java.awt.Container.processEvent(Container.java:2041)
  java.awt.Component.dispatchEventImpl(Component.java:4583)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ This is a redispatch of a new KeyEvent, now on the JButton, while the old will be consumed.
  java.awt.KeyboardFocusManager.redispatchEvent(KeyboardFocusManager.java:1848)
     ^^ It has now found the target Component: The JButton
  java.awt.DefaultKeyboardFocusManager.dispatchKeyEvent(DefaultKeyboardFocusManager.java:704)
  java.awt.DefaultKeyboardFocusManager.preDispatchKeyEvent(DefaultKeyboardFocusManager.java:969)
  java.awt.DefaultKeyboardFocusManager.typeAheadAssertions(DefaultKeyboardFocusManager.java:841)
  java.awt.DefaultKeyboardFocusManager.dispatchEvent(DefaultKeyboardFocusManager.java:668)
     ^^ The retargeting of the KeyEvent begins, by the global KeyboardFocusManager
  java.awt.Component.dispatchEventImpl(Component.java:4455)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Window.dispatchEventImpl(Window.java:2475)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ The initial dispatch, on the Window that holds the JButton.
  java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
     ["EDT pump"]

Here's the stacktrace for F3, which is bound to the JPanel "WHEN_ANCESTOR_OF_FOCUSED_COMPONENT", and fires when either of the buttons (or anything else residing in that JPanel) has focus:

 Thread [AWT-EventQueue-0]
  com.example.ExampleAction.actionPerformed(...)
  javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1636)
  javax.swing.JComponent.processKeyBinding(JComponent.java:2849)
      ^^ This invocation is now on the JPanel and not the JButton anymore
  javax.swing.JComponent.processKeyBindings(JComponent.java:2895)
      ^^ Traversing parents
  javax.swing.JComponent.processKeyEvent(JComponent.java:2812)    <--
     -- same as above

And finally, here's the stacktrace for F4, which is bound to the JButton A "WHEN_IN_FOCUSED_WINDOW", and fires when either of the buttons (or anything else in the entire Window) has focus:

 Thread [AWT-EventQueue-0]
  com.example.ExampleAction.actionPerformed(...)
  javax.swing.SwingUtilities.notifyAction(SwingUtilities.java:1636)
  javax.swing.JComponent.processKeyBinding(JComponent.java:2849)
  javax.swing.KeyboardManager.fireBinding(KeyboardManager.java:267)
  javax.swing.KeyboardManager.fireKeyboardAction(KeyboardManager.java:216)
      ^^ Runs through all (showing) components that have registered WHEN_IN_FOCUSED_WINDOW maps.
  javax.swing.JComponent.processKeyBindingsForAllComponents(JComponent.java:2926)
      ^^ This is a static method, invoking the global KeyboardManager
  javax.swing.JComponent.processKeyBindings(JComponent.java:2918)
      ^^ Given up on local component bindings!
  javax.swing.JComponent.processKeyEvent(JComponent.java:2812)    <--
     -- same as above

It is worth noting that even if a key binding is bound to the InputMap WHEN_IN_FOCUSED_WINDOW for some component, the bound action will only fire if the component is showing and enabled. This means that if you have a JTabbedPane, adding a WHEN_IN_FOCUSED_WINDOW KeyBinding to some component on some tab, the action will only fire if this tab is the one showing - i.e. it won't fire if some other tab of that same JTabbedPane is showing. (The JTabbedPane alters the "visible" property of its contained components when changing tabs, which thus changes the showing property: Showing is when you are visible and all grandparents up to and including the window also are visible). This effect is probably what you want (!), but if not, then you'd obviously just do the key binding on the JTabbedPane instead, or even closer to the root Window.

Also, as could be read out of the JavaDoc rip above, if a component consumes the event, it will not percolate further, which most probably is what you'd want. This means that if the focus is on a JButton, and Space is pressed, this event will be consumed by the button to click it, and the event will not fire any of the WHEN_ANCESTOR_OF_FOCUSED_COMPONENT OR WHEN_IN_FOCUSED_WINDOW Actions bound further up in the hierarchy. Also, if a JTextField is focused, all KEY_TYPED events will be consumed in the same fashion, so that what one writes into the field won't also be processed by Actions higher up in the GUI hierarchy.

However, KEY_PRESSED or KEY_RELEASED events will still not be consumed in the manner described above - which when related to the deviousness of KeyStroke's static construction methods can become a little confusing! (As you probably understand by this, it has at least confused me: Java insisted on firing some key bindings I had added for no-modified characters on a component high up in the hierarchy (some JPanel), for example the key 'i' and 't' and so on, even though a text field way lower in the hierarchy was focused and happily processed what I typed - so that when I wrote some text in the field, for example "He liked it!", I also fired off the global actions!)

It is worth noticing that the older Keymap functionality found on JTextComponents was reimplemented using the InputMap/ActionMap functionality when that was introduced in Java 1.3.

Semantic event firing: To round this off, we'll just do a quick round of the high-level events, or the semantic events as they're also called. Swing is built upon the Model-View-Controller ("MVC") architecture pattern. The details here are way beyond the scope of this article, but I'll do a quick scratch nevertheless (go here for more). First of all, the pattern was apparently invented by a Norwegian (as is object oriented programming too). Now, with the essentials out of the way: Lets start with a button. The View would the physical/visual button as you see it, while the Model is the button's state ("focused", "armed", "pressed" and clicked or on/off), and the Controller is the logic that transitions the model through these states (getting pinged by the view when the user interacted with it).

For Swing, the View and Controller is collapsed into a UI Delegate ("UI"), which forms a part of the Look and Feel ("LaF") system of Swing. This new model is apparently "sometimes referred to" as a Separable Model Architecture. The rationale for this you'll have to pry out yourself from the above-linked documentation - I've never quite understood it..!

You thus end up with three classes for a button: the JButton itself (which is an AbstractButton, which is a JComponent, which is a Component), which has a UI delegate of type ButtonUI, and has a model of type ButtonModel. There is no standard ButtonUI, as this is a part of the installed LaF, but there is a "place to start" called BasicButtonUI. The standard ButtonModel is DefaultButtonModel.

  • You instantiate a JButton. Since it is a JComponent and thus a Component, it can have all the low-level AWTEvents fired on it, if it wants.

  • The JButton constructor instantiates and sets a DefaultButtonModel, and then (effectively) invokes updateUI, which installs the current LnF-dictated ButtonUI.

  • While setting the model, the JButton also installs a bunch of listeners on it. This is so that changes on the model may be reflected physically on the screen by the JButton: If the model becomes pressed, the JButton needs to know so that it can repaint itself into a pressed-looking state.

  • Assuming that we're doing BasicUI (which is Okay, WindowsButtonUI extends BasicButtonUI), the UI installs a slew of listeners on the JButton (which, as mentioned, can have all low-level AWT events delivered to it). It does this by instantiating a BasicButtonListener (a helper for the BasicButtonUI) which implement all these listeners, and then installs this instance by invoking the different add*Listener(instance) methods on the JButton.

  • The BasicButtonListener obviously implements MouseListener.mousePressed and mouseReleased. On pressed, the BasicButtonListener, i.e. the UI delegate updates the model to be armed and pressed, and it furthermore requests focus to the button. ...

  • ... while on release, the BasicButtonListener, i.e. the UI delegate updates the model to be un-pressed.

  • Since the button was armed, this update to un-pressed state makes the model create and fire the ActionEvent!

  • (The BasicButtonListener (i.e. the UI delegate) then also un-arms the model, making it ready for a new round)

As a little side-note, if you press the mouse on the JButton, but then drag the mouse out of the JButton while still holding the mouse button, the hooking of an AWTEventListener in the Container to catch mouse dragged events, (as mentioned above on retargeting of MouseEvents) will make sure that the button is still focused and pressed, BUT the still-fired mouseExit event will make the DefaultMouseListener (i.e. the UI delegate) un-arm the model. So, if you now release the mouse button outside of the JButton, it will not fire.

Anyways, here's a stack trace for a clicked JButton having an ActionListener installed. I've included the AWT Events preceeding this ActionListener invocation:

52729 [AWT-EventQueue-0] AWTEvent: @856d3b java.awt.event.MouseEvent[MOUSE_PRESSED,(57,15),absolute(-305,317),button=1,modifiers=Button1,extModifiers=Button1,clickCount=1] on JButton
53995 [AWT-EventQueue-0] AWTEvent: @681db8 java.awt.event.MouseEvent[MOUSE_RELEASED,(57,15),absolute(-305,317),button=1,modifiers=Button1,clickCount=1] on JButton
53995 [AWT-EventQueue-0] java.awt.event.ActionEvent[ACTION_PERFORMED,cmd=Button with AL,when=1236222620558,modifiers=Button1] on JButton
  com.example.ExampleActionListener.actionPerformed(...)
  javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
  javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
  javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
     ^^ Here we're processing the newly created semantic/high-level ActionEvent
  javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
     ^^ This invocation is setPressed(false), which, since the model was "pressed" and "armed", creates and fires the ActionEvent.
  javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
  java.awt.Component.processMouseEvent(Component.java:6216)
  javax.swing.JComponent.processMouseEvent(JComponent.java:3265)
  java.awt.Component.processEvent(Component.java:5981)
  java.awt.Container.processEvent(Container.java:2041)
  java.awt.Component.dispatchEventImpl(Component.java:4583)
  java.awt.Container.dispatchEventImpl(Container.java:2099)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ Here we're processing the newly created retargeted AWTEvent, on the JButton instance
  java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4556)
  java.awt.LightweightDispatcher.processMouseEvent(Container.java:4220)
  java.awt.LightweightDispatcher.dispatchEvent(Container.java:4150)
     ^^ Invocation of retargeting through LightweightDispatcher
  java.awt.Container.dispatchEventImpl(Container.java:2085)
  java.awt.Window.dispatchEventImpl(Window.java:2475)
  java.awt.Component.dispatchEvent(Component.java:4413)
     ^^ Here we're processing the low-level AWTEvent straight from the operating system, on the Window instance
  java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
     ["EDT pump"]

And with that, we're done!