A few weeks ago a friend asked me about a problem with XMLStreamReader. We have quickly concluded that it is no error at all, it is in the nature of the XML processing tools, but if you encounter it at the first time, it could seem strange. It is about the fact that XML text nodes are not necessarily processed at once, and while you read the XML, you might receive only fragments.

For example if you have the text: "Q&A", which in XML will be escaped to "Q&A", you might end up with reading first the string "Q" then, the "&" and finally the "A", instead of reading it as a whole string. Like the following code:

import java.io.StringReader;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamReader;

public class TestTextNode {
  public static void main(String[] args) throws Exception {
    String xml = "<?xml version=\"1.0\" ?><test>Q&amp;A</test>";
    XMLInputFactory factory = XMLInputFactory.newInstance();
    XMLStreamReader reader = factory.createXMLStreamReader(new StringReader(xml));
    reader.next();
    reader.next();
    System.out.println(reader.getText());
  }
}

On Sun's Java 6 JVM you shall receive just "Q" in the first round. On the consecutive reads, you will receive the other characters, but for the unprepared people, it is just strange. So, why this happens?

XML allows you to have very large files. If you look for example at the wikipedia.org XML dumps, it is not unusual to have XML files larger than a few GB. There is no limit on how big a text node can be, so it is the responsibility of the tool to process it in reasonable chunks. If you order it to load into a DOM, you will receive a large tree in the memory - if you have much more than the XML side itself, you have good chances that it will fit. However on large XMLs or for some kinds of processing, you just stream through the data and do not build a DOM tree.

As in the example above, while you stream though the XML, you will receive TextNodes. These are usually constrained by the:

• closing or other opening tag
• buffer size of the streamer (if it is full, the stream reader will receive the text)
• special escape characters (as above, the escaped & resulted in a new fragment

While the first one is trivial, the second and third is a less-known internal of the XML parsers, but from the memory consumption perspective, it seems it has a good reason behind it.

Now the question remains: are you able to parse the XML and receive all the text consecutive nodes compacted? It depends on the parser, but in Java, you can, just put the following code after the factory initialization:

      factory.setProperty(XMLInputFactory.IS_COALESCING, true);

So it is not magic to change the behavior, although with the recent hardwares and softwares, it might be better to have the coalescing by default, and it could be turned off - although it is definitely fail-safe this way.

When profiling applications, it is always important to measure the time as precise as it can be, and the old way was to measure the system clock with increasing granularity, while in the meantime we have received access to a more precise, thread-specific clock.

In the old-fashioned way, we have two methods to measure the system clock:

• System.currentTimeMillis()
• System.nanoTime()

A few weeks ago there was a question on a Hungarian Java user list about PMD and code review that made me wonder what is the actual state of automated code review or code analyzer tools like PMD compared to the manual process. The political aspects of the code review process brought up some good and some bad memories too...

My quick answer on the mail list was around the following: "I wouldn't call PMD as a code review tool, it is rather a code checker. Even if PMD rules pass (especially for the rules actually make sense) we have had projects where our manual code review improved the quality of the code significantly."

Okay, this is not a big surprise so far. But the reply made me think a little bit: "Our actual development methodology doesn't have any place for code review. I am are looking for way to include it without disrupting the current development timelines. For the first 2-3 projects it will be only feedback, to have some routine in it and later on we can fine-tune the process. However somewhere we shall start and PMD looks good for such."

Good luck for such an initiative! :)

I have some doubts about the long-term success if it will remain feedback only and it is not enforced even in the actual projects. Why I see it that skeptical? I think if something (a) is not mandatory and (b) it is not in the essential culture of the group and (c) it could be skipped to save some time for the hard deadline - will be essentially skipped to save that crucial time... Been there, done that, forgot that... :)

One of our clients did have similar scenario a year ago: a multinational corporation that works mostly with outsourced developers (read: cheaper than everything = not always as qualified as you would like), while we were hired as architects for project assistance. Our first shock was when we've realized that nobody (read: not a single person) has ever required an internal code review of the shipped products. Of course officially they have had it: if the department A of the contractor company developed a product, they have requested a code review from the department B. Guess what: it has passed. Even if they contracted a third party, it was as cheap or much more cheaper than the original one, so there was no surprise they haven't produced any reasonable report. By the way: have you noticed that code review reports make no sense at all for product managers who just know Excel-sheets? :)

Anyway, as architects we were the first to introduce this requirement in the business process, and to be honest, we have made progress in the technical parts, but we have slightly failed the political aspects. At least it reinforced the belief: if it is not mandatory, it will be skipped - period.

In technical terms, we were quite good: we had introduced PMD as an IDE-plugin, so every developer was able to check the rules for itself. Unfortunately PMD contains a lot rules that just make no sense or are irrelevant in the actual project context, so we had to fine-tune the ruleset and eliminate some annoying ones. We have created a few new rules to describe some critical scenario but generally we have left PMD to do the job.

On the political side, we - as architects -  were a bit outsiders: we have had no rights to push the project deadline or block the handover process. Even if the project was buggy (in PMD sense and in our common sense, like String == instead of .equals(...)), we have had no impact on the process. The PMD reports had been sent around, but the management gave it no sh*t (they hadn't understood it, and the deadline was too much worry anyway), so nothing happened with the results. The only thing that successful resonated something was when I've periodically sent a report about the number of errors in the project - with the historical numbers, showing a clearly rising tendency :[. Finally we were allocated with the budget to do some manual code review as well, and it was a clearly better story - well, at least for us, architects :].

Speaking on the good experiences, I've had a job at an investment bank, where we had a clever and responsible team of people around infrastructure libraries. These libraries were crucial, many other departments has used them in production environment - so essentially a solid code review process was in place. There was practically no code that wasn't peer reviewed before make to a new release - with a few exception of emergency releases, but they were in a different branch.

Speaking of our internal developments, we have the luxury of doing such code reviews irregularly, because instead of such process, we rely on:

By the way, this resonates to the "automate everything" motto of successful businesses :)

Cutting a long story short (oh, am I late with that?), my view on the technological part:

As our regular readers already know, Wicket is our favorite web framework and we use it actively in our projects. Wicket is an easy-to-use, well-designed framework and is able to incorporate Ajax in a very nice and easy way. I personally am not a big fan of using Ajax in every corner of the application, however at some points it can make your app much nicer. Let's look at such a case!

Imagine a form where you have to enter possible answers for a survey question, with the number of answers being dynamic. It would be possible to give the user a fixed number of text fields, let's say 8 should be enough, but what happens if the user wants more possible answer? Or what if (s)he only wants 2, and is not really happy about another 6 empty text fields taking up half of the screen? So let's go dynamic and "ajaxify"!

Let's give the user only 2 possible answers to begin with and also a link with the possibily to add more answer rows, and another to remove unused ones. The form can contain a lot of other fields in which we are not interested in, however adding or removing a row should not have any effect on the other fields. What's more the app should keep those values that the user already has entered. Of course the links should be using Ajax and only refresh the dynamic list part of our form page and they should refrain from submitting and/or validating the whole form. If you look up a tutorial or a good book on Wicket it gives you a similar solution:

<!-- DynamicRows.html -->
<form wicket:id="form">
<!-- Other non-repeating fields in the form -->
...
<div wicket:id="rowPanel">
  <span wicket:id="rows">
    <span wicket:id="index">1.</span>
    <input type="text" wicket:id="text"/>
  </span>
  <a href="#" wicket:id="addRow">Add row</a>
</div>
...
</form>

And here comes the associated Java code:

// Relevant constructor code in DynamicRows.java
...
 // Create a panel within the form, to enable AJAX action
 final MarkupContainer rowPanel = new WebMarkupContainer("rowPanel");
 rowPanel.setOutputMarkupId(true);
 form.add(rowPanel);

 // List all rows
 ArrayList rows = new ArrayList(2);
 rows.add(new String());
 rows.add(new String());
 final ListView lv = new ListView("rows", rows) {
   @Override
   protected void populateItem(ListItem item) {
     int index = item.getIndex() + 1;
     item.add(new Label("index", index + "."));

     TextField text = new TextField("text", item.getModel()));
     item.add(text);
   }
 };
 rowPanel.add(lv);

 AjaxSubmitLink addLink = new AjaxSubmitLink("addRow", form) {
   @Override
   public void onSubmit(AjaxRequestTarget target, Form form) {
     lv.getModelObject().add(new String());
     if (target != null)
       target.addComponent(rowPanel);
   }
 };
 addLink.setDefaultFormProcessing(false);
 rowPanel.add(addLink);
...

You can notice, that we have used AjaxSubmitLink for adding a new row into the list. This is needed because the user might already have entered some values in some of the fields and we don't want those to be lost, so the form values have to be submitted. However we would like to avoid getting a validation error in some of the other fields when all we want is to add a new row, so we use addLink.setDefaultFormProcessing(false).

This is a nice solution, however if you try it you'll see that the values entered in the repeating rows get lost, when a new row is added. The reason for this is that after pressing the "Add row" Ajax link the TextFields don't update their backing model (since we have turned off form processing), however the ListView removes and recreates all its TextFields again in its onPopulate() method. And so the "old-new" TextFields will show the original model value.

So what to do now? You can try to update the backing model of all the repeating TextFields in the Ajax "Add row" action, however this is still not a 100% solution in case an invalid value is entered. In case of an invalid value the validation fails, the model doesn't get updated and after recreating the TextFields the invalid value reverts to the last valid value entered. So it turns out that the problem is again caused by the recreation of all the TextFields.

You could cache and reuse all those TextFields in a custom subclass of ListView, as I did for the first time. Or you could browse the source code of ListView and come across a reuseItems property which is just the nice and clean solution to our problem:

  lv.setReuseItems(true);

This will reuse the already created TextFields and will call the populateItem method only for the newly added row. Since the TextFields are reused they will remember the last valid or invalid value entered, and that's what we wanted from the beginning. All the above logic can also be applied to a "Remove row" Ajax action in case it's needed. So, we have made a nice "ajaxified" ListView.

Wicket provides a lot of useful feature, among the others: it provides a lot out-of-the-box components. And if it doesn't suites you, you can easily create your own. Recently we have encountered this with a captcha component: we required a few different features (e.g. easier to read captcha), so we have created our own captcha panel.

  

On the company website, we have indicated the intention to share some of our work as open source. This blog seems to be a good opportunity to start with that process, especially when the toolkit is as simple as our GWT integration solution with Spring framework. (Have you noticed that springframework.org is redirected to springsource.org? I wonder if the packages will be renamed too, just to break backwards compatibility - *evil*)

There was always a fuss about how GWT and Spring can be integrated. There are solutions who go with the hard way: defining the services as servlet paths with some hacking to access the application context, or following the Spring MVC, as separate controllers. A year ago, at the time when the Spring annotation support gained more awareness, there were only a few solution to use that with GWT. At that time, we have developed a GWT application that required a Spring integration, so we took the ideas from Chris Lee's blog, and extended it to our needs. Now, it is our time to share that with the open source community, hoping that we can say something new or at least less-known...

Chris and Martin (check the blogs comments) however started a much more interesting way, which was near perfect for us: define your service interface in GWT, implement it as a normal class on the server side, and let Spring do the magic with the binding and such. You need to have the following:

• GwtServiceHandlerMapping - for mapping the service with a given servlet path
• GwtServiceHandlerAdapter - for processing the request on that url and delegating the request to the invoked method
• RpcProxyWrapper - that handles the client side magic

The company made a good decision in the recent weeks: the target is the sky, but at least the cloud. Amazon AWS offerings are hard to beat, so we have started with that one, played around with different configurations a bit, and finally decided that first we shall migrate the company Subversion repository to the cloud, with ZFS mirrors and encryption.

I'm a long-time fan of the ZFS filesystem and the Sun's OpenSolaris offering around it, basically because this is the best, easily accessible filesystem that provides drive mirroring with checksums, enabling automatic recovery from the underlying storage's failures. So it became a natural plan to run OpenSolaris on EC2, ZFS with EBS volumes mirrored. Although the EBS is meant to be very robust, there are always failures in every system, and we have checked a few blog entries where the EBS actually did fail, so better be prepared...

We know that we cannot achieve absolute secrecy only if we unplug the server, dump it a big hole in a deserted location and forget about it, but it seemed to be reasonable to have some encryption. The plan was that at the time the instance starts, we log in, attach the the encrypted ZFS pool with typing the password. Okay, the running instance may be monitored and the content might be extracted if the infrastructure allows such move, but we hope this is a much harder and more classified job to do, than sniffing around a volume snapshot.

I've mailed to the Sun OpenSolaris EC2 team, and they were very kind giving the initial pointers to look for the stuff. I can recommend the following sites in this topic:

• Immutable Service Container
• Glenn Brunette's Security Weblog
• Encrypting ZFS pools using lofi driver

To cut a long story short, I've used the sun-opensolaris-2009-06/opensolaris_2009.06_32_6.0.img.manifest.xml AMI, and here are the commands that were required to complete the process:

I would like to share some thoughts with you about an untold tale in software engineering. Most probably you have already read or seen quite a few marketing-oriented documents or events with a similar message:

• Use configuration instead of coding
• Change system behavior or business rules without recompiling
• Increased developer productivity...
  

All of these (let's call them dynamics) are nice, however I've never seen (not even in a footer with a tiny font) what is the price you have to pay for these features. And there's sure a price to pay compared to the "old" approach where you didn't put much of your program in configuration, and you had to recompile it for even the smallest changes. Well let's have a closer look at one very common one, the usage of JavaBeans.

A JavaBean is primarily meant to be a "data-holder" (or business object if you like that terminology better), with no or very little application logic included. Their main purpose is to hold data that are set through their setter methods and return those when needed via their getter methods. With some simple rules defined (for an attribute named attribute there must be a getAttribute and a setAttribute method) it is possible to dynamically explore and use any JavaBean with the help of the Java Reflection API (java.lang.reflect package). Well in most of the cases told in the beginning this is the backing concept, and the one that makes a developer's life easier (at least in the development phase). Let's have a look at a simple JavaBean:
This is a very simple JavaBean. If you want to work with it the "old" way, you call those getter and setter methods directly, but then you have to explicitly write those at compile time, making your app not very dynamic. The "new" way is this:
Even better is if you explore the methods and their parameters beginning with "set" and "get" dynamically, and store them in a Map for later use. More or less this is the way all of today's frameworks function, from JSP's Expression Language to Wicket's PropertyModel binding. Since this - not very nice - code can be put in the framework, the developer will see only the nice and easy configuration of ${person.name}="James Bond" or something similar. Ok, but what will the developer see at runtime (with an emphasis on time)?

Well, attached below is a small little tester app, that shows the runtime difference between direct and indirect (reflective) call of a JavaBean's setter and getter methods. The test performs a typical scenario when some dynamically configured behavior of a framework reads and/or updates all the fields of a JavaBean. Please note that the solution used is probably the most simple solution for indirect method invocation, most frameworks use something more complex for example via the java.beans package, but in the end it always comes down to java.lang.reflect.Method.invoke(...)! You can easily run the app yourself, so I only show some interesting results:

Test cycles:	10	50	200	1000	5000	20000	100000	500000
Direct calls (average in ns)	5700	5800	5700	5700	5800	5100	4500	4300
Reflective calls (average in ns)	32000	60000	32000	36000	18000	12000	6300	5100

We can see that there are situations, when it is 10 times slower to call the same method(s) reflectively than directly. This is a huge difference! And the average doesn't even contain the discarded maximum values, which would make the gap even bigger. By increasing the number of test cycles well above 1000, the difference melts down to about 20%. However I think the reason for this is the HotSpot optimization in the JVM when it sees that the same cycle is repeated for an awful lot of time. I am sure that in a normal application where such code is not executed in a cycle like this, JVM optimizations are not likely to be so effective. Most probably in a normal application it can be said that the similar developer-friendly features (dynamics) take up to 5-6 times more cpu time to accomplish for the JVM.
    Well, what do you think: is this a big price to pay for developer productivity?

As I'm preparing to release OKTECH Profiler 1.1, I have checked the performance benchmarks on the profiler itself. It came apparent that the UTF-8 conversion consumes a lot time, so I've started to investigate what happens behind the scenes. I've encountered a little shock at the DataOutputStream class: it has a serious limitation, as it doesn't allows to write strings larger than 64k. I thought those times(*) were over and it is just the Java reference source code that has this limitation, so I've written a small program to double-check it:

Recently I've encountered a problem of displaying comma-separated list items on a web page. It came natural to check if it can be done in CSS or not. This page explains the concept implemented in CSS, and on this example page you can check it yourself. It works in Safari and Firefox, but does not work in IE 7 - it just don't display any comma at all, the items are listed with spaces between them. Too bad :(

That will leave us to implement the comma separated list inside the application, in our case: in Wicket. The following small code fragments explain the basic idea:

We created a container without markup; inside the container we have defined a "comma" component, and a "label" component. If we take a look at the Java code, it is pretty easy to understand how it goes, actual code fragment uses ListView:

   add(new ListView<String>("list", new MyListModel(myParentModel)) {
       private static final long serialVersionUID = 1L;
       @Override
       protected void populateItem(final ListItem<String> item) {
           item.add(new Label("comma", ", ") {
               private static final long serialVersionUID = 1L;
               @Override
               public boolean isVisible() {
                   return item.getIndex() != 0;
               }
           });
           item.add(new Label("label", item.getModel()));
       }
   });

The first comma component is not shown, while all the others are. It is really simple to implement - after you have the idea and design. Of course you might get rid of the <span> elements too, if that makes sense in your application.