Tag Archives: java

OnlineGlom: MySQL support

Of and on over the last couple of weeks, I have added MySQL support to OnlineGlom, like I recently added to Glom itself. Now it works and is in git master. When I have the time I’ll try it out with Google’s Cloud SQL (MySQL) in Google’s App Engine.

As with regular Glom, most of the work was getting the self-hosting tests to work with MySQL – to start the MySQL instances, create the databases, fill them with data, test them and shut them down. The rest of the support was mostly covered already by JOOQ but I had to make sure it always knew what SQL dialect to use.

OnlineGlom: Slightly Saner Logins

The OnlineGlom demo does not require a login. However, the code does let you set up a server that requires a login, and I noticed that a successful login for one person became a login for everybody else. So after the first login, it was as if no login was required for anybody. Yes, really. Of course, this would not do.

So I fixed that, I think, learning some things about Java Servlet sessions along the way. This text is mostly for my own reference, and so that people can tell me how wrong I am, because I’d like to know about that.

In the server-side code

Java servlets already set a JSESSIONID cookie in the browser, but you shouldn’t try to use that cookie to maintain a login across browser sessions. Instead, I now set and get a custom Cookie, in the server code, using javax.servlet.http.Cookie. HttpSession.getId() conveniently provides a unique-enough session ID for me to use in the Cookie. This page about Cookies with GWT seems to suggest setting the cookie in the client-side JavaScript code, using com.google.gwt.user.client.Cookies, but that sounds rather wrong.

I now store the username and password (Yes, that’s not good, so keep reading), associated with the session ID, in a structure that’s associated with the ServletContext, via the javax.servlet.ServletContext.setAttribute() method. I get the ServletContext via the ServletConfig.getServletContext() method. I believe that this single instance is available to the entire web “app”, and it seems to work across my various servlets. For instance, if I login to view a regular page, the images servlet can also then provide images to show in the page. I’d really like to know if this is not the right thing to do.

However, it still stores your PostgreSQL username and password in memory, so it can use it again if you have the cookie from your last successful login. It does not store the password on disk, but that is still not good, because it could presumably still allow someone to steal all the passwords after a breakin, which would then endanger users who use the same password on other website. I cannot easily avoid this because it’s the PostgreSQL username and password that I’m using for login. PostgreSQL does store a hash rather than the plaintext password, but still requires the plaintext password to be supplied to it. I think I’ll have to generate PostgreSQL passwords and hide them behind a separate login username/password. Those passwords will still be stored in plaintext, but we won’t be storing the password entered by the user. I’d like to make this generic enough that I can use other authentication systems, such as Google’s for App Engine.

To avoid session hijacking, I made the cookie “secure”, meaning that it may only be provided via a secure protocol, such as HTTP. I believe this also means that client (javascript) code is not allowed to read it, so it can only be read by the server via HTTP(S). I did that with the javax.servlet.http.Cookie.setSecure() method, though I had to make a build change to make that available.

The login servlet now checks that it has been called via HTTPS, by using the ServletRequest.isSecure() method, and uses HTTPS when testing via mvn gwt:run. It refuses to do any authentication if HTTPS was not used, logging an error on the server.

In the client side code

I added a check for what protocol is used. If it’s not https then I warn that login cannot work. This does not add any security, but it’s a helpful hint.

Actually, the entire site must therefore be served via HTTPS, not just the login page, or we would violate the Same Origin Policy by mixing protocols, which the browser would rightfully complain about. At this point I noticed that most serious sites with logins now use HTTPS for their entire site. For instance, Google, Amazon, Facebook. This seems like a good simple rule, though I wonder if many projects don’t enforce it just to make debugging easier.

I also converted the popup login dialog into a proper login page, making sure that it takes the user to the desired page afterwards.

Online Glom is now all Java

Over the last couple of weeks I have reimplemented just enough of the C++ libglom code as Java in Online Glom‘s gwt-glom, removing the need for java-libglom (which wrapped libglom for Java via SWIG). It’s now working and deployed on the Online Glom test server.

This makes both development and deployment much easier. It also made the source code all camelCase so it’s not offensive to the eyes of Java coders.

To replace libglom’s use of GdaSqlBuilder, I used jOOQ. That worked well, thanks to its maintainer, Lukas Eder, who was very helpful and who quickly added some API that I needed.

Now that the code is all Java I really hope that more people will look over the code and point out anything that can be improved. I still don’t know Java like I know C++ so please don’t be shy about telling me that I have made mistakes.

GWT, Javascript, and serialization

Removing the use of java-libglom let me simplify the code, because I can now send object, such as LayoutItem, to the client without needing to copy it into a separate LayoutItemDTO object that existed just because the original wasn’t serializable, so it could be sent from the server (Java) to the client (JavaScript, compiled from Java).

However, this raised some new issues. I wanted some of the objects to contain some extra cached data, so that the client code did not have to calculate it itself, often by retrieving some other related object. Right now these are extra member variables in the classes, but that prevents me from splitting the code off into a new java-glom library.

Furthermore, any class that is sent between the client and server must fully conform to the requirements of the GWT Java-to-JavaScript compiler, even if that method will not actually be run on the server. For instance, I tried to add clone() methods, for use on the server, but that broke the JavaScript compilation because it doesn’t have an equivalent for Object.clone().

Those restrictions on the Java code that is allowable on the client side (because it will be compiled to JavaScript) were particularly awkward when the compiler (or Maven, or something) refused to give me clues about what was wrong. For instance, it took me 2 frustrating days to fix this small error by breaking the code apart until only the problem code remained. At other times, there was no real error on stdout, but there were clues (in a variety of hard-to-read formats) in the HTML generated by mvn site. Or sometimes, I could see errors when building inside Eclipse, but not outside.

The GWT system works great, but something is inconsistent about how it shows errors, and it can’t be right that some compilation errors only show up when running, rather than when building.

Next steps

As much as I would like to move on to implementing editing, I need to spend some time now on getting some regression tests set up in the maven build. These must create and run temporary PostgreSQL database instances like I do in the Glom autotools build. For instance, I need to check that the new SQL-building code, using jOOQ, is really safe from SQL injection like the libglom code seems to be.

My recent changes also caused the OnlineGlomService async API to be particularly inefficient, sometimes sending far more data back to the server than should be necessary. I will try to avoid that and try to make this API smaller, to avoid so many round trips.

Online Glom: Reports

Over the last few weeks, in occasional spare moments, I have added report generation to Online Glom. It kind of works now, using the regular report definitions from the existing .glom files, and generating reports that look much like the reports in the desktop Glom version. I deployed this at onlineglom.openismus.com/OnlineGlom/ .

I used JasperReports for this, writing some code to map Glom’s report structure to the JasperReports structure. I chose JasperReports mostly based on its popularity. There are various tools and libraries that use it and the JasperSoft company seems to be very successful by supporting it.

However, I will probably replace JasperReports with some custom code instead, because:

  • The JasperReports API is almost completely undocumented. I figured out what to do only by looking at various snippets of code from random places. The project seems to focus on the use of visual tools to build report designs, rather than the use of a generic API by such tools.
  • JasperReports demands absolute positioning, with no automatic layout, and that will never make sense for HTML.

I will try the DynamicJasper wrapper/helper before abandoning JasperReports, but I don’t see how it can avoid the fundamental problem of absolute positioning.

Online Glom: More Translation

Online Glom’s standard UI strings are now translated too, instead of just the strings that are in the .glom files. I added some initial translation files (mentioned here too) but I need people to translate them, please. Feel free to just email the file to me. I am tempted to add them to the desktop UI’s translations so I can copy them across.

I also changed it from using a lang= token in the URL to using GWT’s regular locale= query value, re-simplifying much of my previous Online Glom code to support translations.

Now that I see how each new translation adds another set of gwt-java-to-javascript permutations to the already-slow build, so it now builds 41 permutations, I might switch later from the Constants to the Messages technique for GWT translation, because I don’t think that string lookup will be a big performance problem.

Online Glom: Now easy to build

Building gwt-glom

At least on Ubuntu, it’s now easy to build and test gwt-glom. You can just do:

$ sudo apt-add-repository ppa:openismus-team/openismus-glom-unstable
$ sudo apt-get install default-jdk maven2 libjava-libglom-java glom-utils
$ git clone git://gitorious.org/online-glom/gwt-glom.git
$ cd gwt-glom
$ mvn gwt:run

This opens the GWT Development Mode GUI, which serves gwt-glom via jetty and lets you see it in your browser.

Chrome is the most likely to have the gwt plugin that you need for development mode, though it’s available for some versions of Firefox.

How we made it this easy

GWT projects, like other Java projects, typically use maven (mvn) for their build system. maven is nothing like autotools.

Maven usually downloads dependencies automatically, either from the central maven repository, from some (maybe private) other maven repository that you specify. This sounds unstable, but you specify exact version numbers, so you can be sure that your project will continue to build. So maven doesn’t have the separation of building and packaging that I’m used to in the C/C++/autotools world. It feels odd to me, but I’m going with the flow.

However, java-libglom uses JNI to provide a Java API around a C++ (libglom) API, so it uses both Java (architecture-independent) and C++ (compiled and linked for particular architectures).

java-libglom installs a native shared libary. We packaged that for Ubuntu in our Glom PPA (stable and unstable) as libjava-libglom-java so you can install it easily.

java-libglom also creates .pom and .jar files (for the API, the sources, and the javadoc). These are in the central maven repository so maven can just download the .jar.

That libjava-libglom-java Ubuntu package also installs the .pom and .jar files that maven needs, but those are only useful for mvn-debian, which is apparently only useful for building other Debian/Ubuntu packages. There is no apparently no way to using mvn-debian’s local repository while also using the central maven repository for other stuff.

Java with autotools

By the way, java-libglom uses autotools, although the autotools Java support is barely useful and limiting, so we have custom rules for most stuff. However, autotools does let us generate the Java and C++ files from swig, and build the native shared library. That seemed harder to do with maven, though maven would have made it easier to deal with the generated Java code.

I do like how maven just defaults to using your .java files, and test ,java files properly if you put them in the correct places. For instance, see the maven quickstart project.

Online Glom: Translations

This week I changed libglom’s TranslatableItem API to allow Online Glom to use the recent translations of the Glom example files. This requires the latest unstable version of libglom-1.22 and java-libglom.

There is now a language drop-down list at the top-right, and that is shown in the URL as, for instance, &lang=de (I might change that to locale instead of lang). Choosing a different language will change the table titles, field titles, group titles, etc, to the chosen language.

I have deployed that to the Online Glom test server. For instance:

I have not yet done the same for the translatable strings that are in gwt-glom’s Java source code, such as “Search”, and the “Open” and “Details” buttons. I’m not much looking forward to the awkward way that it should apparently be done. gettext’s _() macro is much cleaner.

Online Glom: Implementing a search box

I just implemented my first feature in Online Glom after taking over from Ben Konrath. Luckily I could cargo-cult his work, with some help from Eclipse’s code navigation and refactoring. I added a text box for an easy full text search of the current table, to filter the rows shown. For instance, you can filter the list of packages here.

I’m proud of myself, but I can’t help feeling that it needed far too many code changes, in far too many interconnected classes, just to add a text box in the browser and then use the resulting text to slightly change a method call on the server. I took the time to write a detailed ChangeLog/commit-message just to be sure that I understood what was happening.

This is apparently how things are in the land of Java. In its defence, there are reasons for the abstractions and separations. For instance, OnlineGlom code does the GWT thing of allowing for different View implementations for different clients, such as mobile or desktop, but it only has one type of Views for now. It also uses a Model/View/Presenter architecture, via its Activities, which makes it possible to test more code logic without getting the UI involved. And there’s the Places idea, which maps history tokens (parts of URLs) to Activities and vice-versa.

I wish there were some way to have the useful architecture without the repetitive code that gets in the way of the interesting stuff.

Online Glom

Ben Konrath recently finished up a first real milestone in Online Glom development, and I have deployed it, with the example Glom files, on an Amazon EC2 instance. Online Glom is still a read-only UI, so you can’t edit data in the database, but I think it’s already useful for some situations. Well, I do want to add the quick search feature soon to really finish off the read-only functionality. Report-generation would be good too.

The Film Production Manager example is a fairly good example of the complex systems that Glom can support, without SQL and without programming.

Here’s a screenshot of the same thing in the desktop UI.

I have also deployed my old Debian Repository Analyzer, full of real data, so you can get a feel for navigating around the related records. I spent lots of time updating that for the latest python-apt API and libgda (with python) APIs, and it’s now in a gitorious project. Many thanks to Michael Vogt for his help with python-apt.

Now that David King has packaged java-libglom (as libjava-libglom-java) for Ubuntu Oneiric in the Openismus PPA, it’s really easy to work with the gwt-glom code on that distro.

What’s Next

This proves that GWT was a sane choice, though development has not been as quick as I’d hoped . But I don’t think a generic framework can ever be developed as rapidly as most data-driven web sites that start as quick hacks. The difficulties have shown that I was right to focus on a restricted set of functionality at first. You can get a sense of how development has progressed by looking at the gwt-glom commit log, though there’s lots of work in java-libglom too.

I now plan to take the development further myself. That’s a nice way to get more deeply reacquainted with Java and web development. It’s easier to hack on the project at this stage, before it gets huge, rather than trying to do this from scratch. Ben Konrath has already made the large architectural decisions so that I don’t have to. I’m even enjoying Eclipse, which is a much more pleasant experience with Java than with C++. Well, using the Eclipse IDE is still like shopping in a flea market but with Java you will quickly find useful things.

I plan to do things roughly in this order:

  • Add the Quick Find feature, for searching. Update: Done
  • Add report generation.
  • Add print layout printing.
  • Maybe investigate how to make theming via CSS easier.
  • Allow editing of data. This will be a big task.

At the same time, I will play with avoiding the need for java-libglom’s JNI binding to the libglom C++ library. I would need to reimplement Glom document parsing in Java, but that would be easy as it’s just XML. But I would also need some replacement for GdaSqlBuilder, to build SQL queries without manually concatenating and escaping text.

 

Learning Java Web Stuff

Relearning Java and JSP

Over the last couple of weeks I’ve been relearning Java and JSP and learning about Google’s GWT . I learnt Java when it was first released but I last used it around six years ago and I never used it seriously in a  large project.

I’m now up to speed on new stuff in the Java language, such as annotation, generics, and enumerated types, some of which I really missed not having before. A newer edition of Bruce Eckel’s Thinking In Java was great for that. I’ve always  liked how he includes comparisons to other programming languages (C++, Python, C#) and is unafraid to criticize the API that he’s demonstrating, though I wonder how interesting that is to other people.

For JSP I tried a different style of book: Head First Servlets and JSP. It’s full of clip art and silly captioned kung-fu photos and cartoons but that really did help me to keep reading. I’m only slightly ashamed.

No, I’m not going to do less C and C++. I love C++. It’s just not what’s used for web development.

Learning GWT

Of course any web site these days must use AJAX to provide a more interactive page, exchanging data with the server and updating the page without doing a whole page refresh. I’d like to avoid writing or maintaining Javascript if at all possible, so Google Web Toolkit seems like a good choice. Bizarrely, it generates javascript from Java, and it even has (imperfect) Javascript implementations of some of the standard Java library. It’s weird but it seems to work. I believe it’s used for gmail and Google Maps.

I’m using the GWT in Action book, though the online GWT documentation seems fine, and the self-hosting runnable examples are really helpful.

Maven

This also included learning about the maven and ant build/configuration tools. maven feels roughly equivalent to autotools plus pkg-config, with ant being roughly equivalent to make. As far as I can tell maven is the in thing.

So far maven is working out well for me, though all the XML to use the necessary plugins feels like voodoo and it has taken me days to get some plugins working, for instance to build WAR files to deploy to tomcat, and to use JNI. The incredibly obscure error messages don’t help and when it does what I want I am thankful but never quite sure how it knew what I wanted.

I also feel weird about using plugins from sites that I’ve never heard of. Without being familiar with the history and community, I have no way to know what is the official best plugin to solve particular problems.

Eclipse Pain Again

This is an opportunity to give Eclipse another chance. After all, it’s meant to be wonderful for Java development, right?

But I seem to need various extras to work with common things like JSP and maven, which forces me to use the hateful “Software Updated and Add-ons” feature. The problems with this are numerous:

  • It shows me many similarly but incomprehensibly named add-ons so I don’t know which one I want. It should explicitly tell me exactly what I would expect to see in my Eclipse UI after installing each thing.
  • It sometimes shows me duplicates of add-ons, just in slightly differently-named directories. What do I choose?
  • It sometimes shows me countless minor versions of the same add-on, instead of just showing me the latest one. There’s a checkbox for that in some versions – it should be on by default.
  • It almost always tells me that some dependency (or some version of some dependency, I guess) is missing. If it doesn’t know how to find it then I certainly don’t. Fail.

I managed to install the WebTools add-on in one of my Eclipse installations. But it has a rather hacky MSWindows-centric wizard, which assumes that all of Tomcat is installed in one directory and won’t let me get further until that is true. Of course on Linux distros things are split up between /usr/lib, /usr/etc, etc.  Some helpful Fedora people seem enthuasiastic about fixing it.

Anyway I generally dislike the idea that I must use a particular project wizard for things that are orthogonal. I shouldn’t be forced to choose between my programming language, my build system, or some of my dependencies. They should be independently interchangable. I suspect that I’d be happy with Eclipse just as an editor, using the command-line with my hand-edited build files, but I never seem to get that far. Eclipse should make it more obvious how to do that because it’s currently a rather hidden feature in some of the new-project wizards.

Note that I’ve tried standard Eclipse in Ubuntu Intrepid (Even Jaunty only has Eclipse 3.2), Eclipse 3.4 in Jaunty (by downloading it and running it from a directory, which works surprisingly well), and “Fedora Eclipse” 3.4  in Fedora 10 in a vmware image.

I’m not ready to give up just yet. I’ll take another run at using Eclipse just as a simple editor, trying not to use any special add-ons. But my patience surprised me.

Online Glom

This isn’t just to refresh these skills and prove to myself that I can still learn. I plan to use this to create a web UI for Glom. My plan is roughly:

  • Use JSP for the standard static parts of the page. This gives me easy authentication and session management.
  • Use GWT from within JSP to construct large parts of the page (identified by div tags). This gives me fancy AJAX UI widgets and a way for client-side code to communicate with code on the server without a page refresh.
  • On the server side, use my C++ libglom library via a Java wrapper generated with SWIG. This will at least give me database structure and UI layout details from .glom documents.

Although AJAX allows us to do more than the old submit-form/get-new-page UI of CGI, I am still not looking forward to dealing with the increased use of async coding compared to desktop coding, batching information up to reduce client/server communication, and being restricted to aggregations of serializable/copy-by-value primitive types when doing that.

My initial inept attempts at all this are in my online_glom repository on github. There’s hardly anything to see but there’s probably already plenty to correct.