Grails on Google App Engine – Part 3: Persistence

October 17, 2010 at 5:10 pm | Posted in GAE, Grails, Groovy, Java, Web | 4 Comments
Tags: , , , , , , , ,

The App Engine plugin installation script will prompt you for the persistence option you want to choose in your project. Google lets you select only one of its persistence implementations for BigTable: JDO or JPA. Using plain GORM is not an option. I decided to go with JDO for the simple fact that there’s more documentation for JDO than for JPA. Despite the fact that there’s documentation it certainly doesn’t cover all the questions I had. I found myself digging for answers to undescriptive exception messages for hours and hours in forums. Implementing persistence for my relatively easy domain model by far took the most of my development time making it a very frustrating experience. There are some good blogs about JDO persistence in App Engine that might be helpful to you: Google App Engine Java Persistence, Google App Engine Java Experiments. Often times I could only make things work by trail and error especially when it comes to modeling relationships.

Domain Classes

Grails domain classes have to reside in grails-app/domain and must be placed in a package. Only then the DataNucleus tool will enhance your classes. App Engine introduces its own data types. You definitely want to have a look at the list before you define the data types to your fields. It’s really a hassle to change the data type after you deployed your application to production and want to change the data type. Usually that means that you have to run a data migration script because App Engine might throw an exception on you for existing data. In my application I used the data type com.google.appengine.api.datastore.Key as primary key. The class Key has a String representation that you can use if you want to send it to your client code. This String representation can be converted back to a Key. The class you use for that is com.google.appengine.api.datastore.KeyFactory.

String bookmarkId = KeyFactory.keyToString(bookmark.key)
Key key = KeyFactory.stringToKey(bookmarkId)

You can try to exclude the default domain object fields id and version. However, that doesn’t quite propagate down to App Engine. The DataStore Viewer still shows both fields. I am not sure if there’s a different way around it.

static mapping = {
   id visible: false
   version visible: false
}

Defining relationships between domain objects can be tedious. More than often you run into funny error messages trying to store or retrieve data. You got to read the App Engine manual very closely to make them work. Especially when it comes down to defining many to many and self-referential owned relationship it can get hairy. Often times I had to fall back to just reference the Key of a different domain class to create a relationship.

Service Classes

The App Engine plugin automatically configures and provides you with Spring’s JdoTemplate that can be injected into your Grails services. Transaction boundaries can be set by wrapping your database interactions into a transactionTemplate. The JdoTemplate is easy to use and shouldn’t be new to you if you used one of Spring’s template implementations before like the JdbcTemplate. In some cases I had to fall back to the JDO low-level API because the JdoTemplate doesn’t provide the functionality. For my needs I created a super class that injects the JdoTemplate and the transactionTemplate.

class JdoService {
   def jdoTemplate
   def transactionTemplate
}

By default the JDO persistence configuration (jdoconfig.xml) created by the App Engine plugin auto-creates transactions. If you want to have full control over your transactions use the transactionTemplate. When you want to operate on more than one domain object within a transaction you have to set the property datanucleus.appengine.autoCreateDatastoreTxns to false.

<?xml version="1.0" encoding="utf-8"?>
<jdoconfig xmlns="http://java.sun.com/xml/ns/jdo/jdoconfig"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://java.sun.com/xml/ns/jdo/jdoconfig">

    <persistence-manager-factory name="transactions-optional">
        ...
        <property name="datanucleus.appengine.autoCreateDatastoreTxns" value="false"/>
    </persistence-manager-factory>
</jdoconfig>

Using the JdoTemplate doesn’t require you to close your connection. This is being handled by Spring. This is an example on how I used it:

def findAllByUser() {
   def bookmarks = []

   transactionTemplate.execute( { status ->
      def user = getLoggedInUser()
      bookmarks = jdoTemplate.find(Bookmark, "userKey == userKeyParam", "com.google.appengine.api.datastore.Key userKeyParam", [user.key] as Object[], "created asc")
   } as TransactionCallback)

   bookmarks
}

Despite the benefits of JdoTemplate you sometimes have to leverage the low-level API. However, you don’t have to create a class to retrieve the PersistenceManagerFactory. You can easily get it from the JdoTemplate. Make sure you close your resources after you used them. I wrote a little helper class to does that. One of the reasons I had to use the low-level API was to implement pagination. I chose to go down the naive, classic path of counting all records of a specific domain object and then retrieve the a list of these objects for the page we want to display. Using the range operator has implications for performance: the datastore must retrieve and then discard all results prior to the starting offset. For example, a query with a range of 5, 10 fetches 10 results from the datastore, then discards the first 5 and returns the remaining 5 to the application. I don’t expect users to have thousands and thousands of bookmarks in one folder so this would be a minor issue. Furthermore, I wanted to get up and running as soon as possible. Alternatively, this posting describes a performant but harder to implement solution.

def findAllByUserInRange(offset, max) {
   def bookmarks = []
   def total

   transactionTemplate.execute( { status ->
      def user = getLoggedInUser()
      Query query

      try {
         final filter = "userKey == userKeyParam"
         final parameters = "com.google.appengine.api.datastore.Key userKeyParam" 
         query = jdoTemplate.getPersistenceManagerFactory().getPersistenceManager().newQuery(Bookmark)
         query.setFilter(filter)
         query.declareParameters(parameters);
         query.setResult("count(this)")
         total = query.execute(user.key)
         query = jdoTemplate.getPersistenceManagerFactory().getPersistenceManager().newQuery(Bookmark)
         query.setFilter(filter)
         query.declareParameters(parameters);
         query.setOrdering("created asc")
         query.setRange(offset, offset + max)
         bookmarks = query.execute(user.key)
      }
      finally {
         JdoUtil.closeAll(query)
      }
   } as TransactionCallback)

   new Page(offset, max, total, bookmarks)
}

Indexes

The App Engine datastore maintains an index for every query an application intends to make. As the application makes changes to datastore entities, the datastore updates the indexes with the correct results. When the application executes a query, the datastore fetches the results directly from the corresponding index. As this might hold true for your development machine this isn’t necessarily true when you deploy your application to App Engine. Often times App Engine didn’t indicate that I have a missing index. You either have to analyze your queries very closely and add the indexes manually in grails-app/conf/datastore-indexes.xml or you’ll find yourself seeing exceptions. These exception can be fixed easily. However, you have to redeploy your application. It might take App Engine a while (sometimes 30 minutes or longer) to build the indexes once you have them defined. Indexes that are in the process of being built cannot serve the query causing an exception. This makes it impossible to deploy a new index without problems for your users. At the moment there’s no way to delete indexes over the administration console.

More on this topic:

Advertisements

4 Comments »

RSS feed for comments on this post. TrackBack URI

  1. […] Grails on Google App Engine – Part 3: Persistence […]

  2. […] Grails on Google App Engine – Part 3: Persistence […]

  3. […] Grails on Google App Engine – Part 3: Persistence […]

  4. […] Grails on Google App Engine – Part 3: Persistence […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.
Entries and comments feeds.

%d bloggers like this: