Tag Archives: OnlineGlom

OnlineGlom: MySQL support

Of and on over the last couple of weeks, I have added MySQL support to OnlineGlom, like I recently added to Glom itself. Now it works and is in git master. When I have the time I’ll try it out with Google’s Cloud SQL (MySQL) in Google’s App Engine.

As with regular Glom, most of the work was getting the self-hosting tests to work with MySQL – to start the MySQL instances, create the databases, fill them with data, test them and shut them down. The rest of the support was mostly covered already by JOOQ but I had to make sure it always knew what SQL dialect to use.

OnlineGlom: Slightly Saner Logins

The OnlineGlom demo does not require a login. However, the code does let you set up a server that requires a login, and I noticed that a successful login for one person became a login for everybody else. So after the first login, it was as if no login was required for anybody. Yes, really. Of course, this would not do.

So I fixed that, I think, learning some things about Java Servlet sessions along the way. This text is mostly for my own reference, and so that people can tell me how wrong I am, because I’d like to know about that.

In the server-side code

Java servlets already set a JSESSIONID cookie in the browser, but you shouldn’t try to use that cookie to maintain a login across browser sessions. Instead, I now set and get a custom Cookie, in the server code, using javax.servlet.http.Cookie. HttpSession.getId() conveniently provides a unique-enough session ID for me to use in the Cookie. This page about Cookies with GWT seems to suggest setting the cookie in the client-side JavaScript code, using com.google.gwt.user.client.Cookies, but that sounds rather wrong.

I now store the username and password (Yes, that’s not good, so keep reading), associated with the session ID, in a structure that’s associated with the ServletContext, via the javax.servlet.ServletContext.setAttribute() method. I get the ServletContext via the ServletConfig.getServletContext() method. I believe that this single instance is available to the entire web “app”, and it seems to work across my various servlets. For instance, if I login to view a regular page, the images servlet can also then provide images to show in the page. I’d really like to know if this is not the right thing to do.

However, it still stores your PostgreSQL username and password in memory, so it can use it again if you have the cookie from your last successful login. It does not store the password on disk, but that is still not good, because it could presumably still allow someone to steal all the passwords after a breakin, which would then endanger users who use the same password on other website. I cannot easily avoid this because it’s the PostgreSQL username and password that I’m using for login. PostgreSQL does store a hash rather than the plaintext password, but still requires the plaintext password to be supplied to it. I think I’ll have to generate PostgreSQL passwords and hide them behind a separate login username/password. Those passwords will still be stored in plaintext, but we won’t be storing the password entered by the user. I’d like to make this generic enough that I can use other authentication systems, such as Google’s for App Engine.

To avoid session hijacking, I made the cookie “secure”, meaning that it may only be provided via a secure protocol, such as HTTP. I believe this also means that client (javascript) code is not allowed to read it, so it can only be read by the server via HTTP(S). I did that with the javax.servlet.http.Cookie.setSecure() method, though I had to make a build change to make that available.

The login servlet now checks that it has been called via HTTPS, by using the ServletRequest.isSecure() method, and uses HTTPS when testing via mvn gwt:run. It refuses to do any authentication if HTTPS was not used, logging an error on the server.

In the client side code

I added a check for what protocol is used. If it’s not https then I warn that login cannot work. This does not add any security, but it’s a helpful hint.

Actually, the entire site must therefore be served via HTTPS, not just the login page, or we would violate the Same Origin Policy by mixing protocols, which the browser would rightfully complain about. At this point I noticed that most serious sites with logins now use HTTPS for their entire site. For instance, Google, Amazon, Facebook. This seems like a good simple rule, though I wonder if many projects don’t enforce it just to make debugging easier.

I also converted the popup login dialog into a proper login page, making sure that it takes the user to the desired page afterwards.

Glom: Experimental MySQL Support

Glom uses PostgreSQL and doesn’t try to offer the user a choice of anything else. That’s because it does what Glom needs, there’s no need to confound the user with an incomprehensible choice, and I’ve no wish to maintain multiple sets of code. It’s hard enough keeping up with changes in PostgreSQL, though Glom’s regression tests help.

However, I played around with adding MySQL support as a build-time alternative via the –enable-mysql configure option. The basic stuff now works both in the UI and in the regression tests. Those tests can now run each self-hosting database test with all 3 backends.

This is mostly just so I could learn about MySQL, so I can reimplement it in Java for OnlineGlom. That would let me use Google’s Cloud SQL, which is based on MySQL. The main work has been figuring out how to initialize a MySQL database store on disk and then start and stop MySQL instances. It’s even more funky than with PostgreSQL. I did need an addition to libgda to support non-standard MySQL port numbers but, as usual, Vivien Malerba fixed that for me quickly. There’s also the rather huge problem that AppArmor on Ubuntu prevents us from starting MySQL with anything but the standard database data, and we can’t expect the user to go editing AppArmor config files. At least with MySQL 5.6, it should be possible to start a MySQL instance without it having no password for a few seconds, as is necessary with MySQL 5.5. I need to start and stop custom instances so I can run tests automatically.

I’ve committed it to Glom’s git master, and its in the 1.23.3 release, just in case anyone wants to improve it. There are some TODO_MySQL comments in the tests where we expect something to fail with MySQL. For instance, I have not added support for editing the MySQL users and groups. And there are likely to be problems with keeping data when changing field types, which doesn’t seem to be tested thoroughly for any backend. libgda is also missing some support for binary field types, needed for Glom’s image fields.

Over the years, various people have complained about Glom not using MySQL. Here is your chance to actually work on that, with tests to show if your work is enough.

Online Glom: Image fields

I found some work in one of my old branches and cleaned it up, so now OnlineGlom supports image fields too.

Online Glom with an image field

As usual, it was far more work than seemed necessary. GWT’s Image widget is not much more than a wrapper around the HTML <img> tag, so I had to create a separate service, with the same authentication system, to serve image data and invent a URL syntax to refer to the images from the database. It is certainly easier with GTK+ code on the desktop, even when delivering the image data asynchronously. This feels like something that a web progamming system should take care of, even if this is what happens behind the scenes. I wonder if any do.

Next, I want to make sure that OnlineGlom can handle tables whose primary keys are not numeric, because we’ve been hard-coding that in a few places. Then I hope I can start the big job of supporting data editing.

Online Glom is now all Java

Over the last couple of weeks I have reimplemented just enough of the C++ libglom code as Java in Online Glom‘s gwt-glom, removing the need for java-libglom (which wrapped libglom for Java via SWIG). It’s now working and deployed on the Online Glom test server.

This makes both development and deployment much easier. It also made the source code all camelCase so it’s not offensive to the eyes of Java coders.

To replace libglom’s use of GdaSqlBuilder, I used jOOQ. That worked well, thanks to its maintainer, Lukas Eder, who was very helpful and who quickly added some API that I needed.

Now that the code is all Java I really hope that more people will look over the code and point out anything that can be improved. I still don’t know Java like I know C++ so please don’t be shy about telling me that I have made mistakes.

GWT, Javascript, and serialization

Removing the use of java-libglom let me simplify the code, because I can now send object, such as LayoutItem, to the client without needing to copy it into a separate LayoutItemDTO object that existed just because the original wasn’t serializable, so it could be sent from the server (Java) to the client (JavaScript, compiled from Java).

However, this raised some new issues. I wanted some of the objects to contain some extra cached data, so that the client code did not have to calculate it itself, often by retrieving some other related object. Right now these are extra member variables in the classes, but that prevents me from splitting the code off into a new java-glom library.

Furthermore, any class that is sent between the client and server must fully conform to the requirements of the GWT Java-to-JavaScript compiler, even if that method will not actually be run on the server. For instance, I tried to add clone() methods, for use on the server, but that broke the JavaScript compilation because it doesn’t have an equivalent for Object.clone().

Those restrictions on the Java code that is allowable on the client side (because it will be compiled to JavaScript) were particularly awkward when the compiler (or Maven, or something) refused to give me clues about what was wrong. For instance, it took me 2 frustrating days to fix this small error by breaking the code apart until only the problem code remained. At other times, there was no real error on stdout, but there were clues (in a variety of hard-to-read formats) in the HTML generated by mvn site. Or sometimes, I could see errors when building inside Eclipse, but not outside.

The GWT system works great, but something is inconsistent about how it shows errors, and it can’t be right that some compilation errors only show up when running, rather than when building.

Next steps

As much as I would like to move on to implementing editing, I need to spend some time now on getting some regression tests set up in the maven build. These must create and run temporary PostgreSQL database instances like I do in the Glom autotools build. For instance, I need to check that the new SQL-building code, using jOOQ, is really safe from SQL injection like the libglom code seems to be.

My recent changes also caused the OnlineGlomService async API to be particularly inefficient, sometimes sending far more data back to the server than should be necessary. I will try to avoid that and try to make this API smaller, to avoid so many round trips.

Online Glom: Deployment

Today I deployed the latest Online Glom (gwt-glom) on a new Amazon AWS instance of Ubuntu Precise, again connecting Apache with Tomcat. This time I took notes about exactly how I did it. I wonder if this is something that I should put in a Puppet configuration (I have never used Puppet).

It took me a while to figure this out last time, but now it’s clearer to me. However, I still don’t know how to avoid the need for the /OnlineGlom/ suffix in the URL.

Online Glom: Reports

Over the last few weeks, in occasional spare moments, I have added report generation to Online Glom. It kind of works now, using the regular report definitions from the existing .glom files, and generating reports that look much like the reports in the desktop Glom version. I deployed this at onlineglom.openismus.com/OnlineGlom/ .

I used JasperReports for this, writing some code to map Glom’s report structure to the JasperReports structure. I chose JasperReports mostly based on its popularity. There are various tools and libraries that use it and the JasperSoft company seems to be very successful by supporting it.

However, I will probably replace JasperReports with some custom code instead, because:

  • The JasperReports API is almost completely undocumented. I figured out what to do only by looking at various snippets of code from random places. The project seems to focus on the use of visual tools to build report designs, rather than the use of a generic API by such tools.
  • JasperReports demands absolute positioning, with no automatic layout, and that will never make sense for HTML.

I will try the DynamicJasper wrapper/helper before abandoning JasperReports, but I don’t see how it can avoid the fundamental problem of absolute positioning.