Simple CRUD with Gaelyk

October 21, 2010 at 7:26 pm | Posted in GAE, Groovy, Search | 5 Comments
Tags: , , , , , , ,

After having written a Grails application for Google App Engine I know exactly what the pain points are in production: Cold startup times make using the application an unenjoyable experience. App Engine calls for a lightweight application with far less JAR baggage and fast startup times. Yesterday I attended Guillaume LaForge’s session “Gaelyk, a lightweight Groovy toolkit for Google App Engine” at SpringOne2GX presenting the nuts and bolts of Gaelyk. The session was reason enough to think about rewriting my application to Gaelyk. To get started I whipped up a simple CRUD example in Gaelyk. All you need to do is to download the Gaelyk template application and you are ready to go. The test application will have four simple use cases: show all entities of a certain type (in my case a “Goal” entity), create new ones, update and delete them.

Gaelyk doesn’t support multiple closures in a controller class to automatically expose URLs by convention like Grails does. Instead you have to define your URL routing for each operation separately.

war/WEB-INF/routes.groovy:

get  "/goals",                forward:  "/showGoals.groovy"
get  "/goals/add",            forward:  "/WEB-INF/pages/goal.gtpl"
post "/goals/insert",         forward:  "/insertGoal.groovy"
get  "/goals/delete/@id",     forward:  "/deleteGoal.groovy?id=@id"
get  "/goals/edit/@id",       forward:  "/editGoal.groovy?id=@id"
post "/goals/update",         forward:  "/updateGoal.groovy"

To show all Goals I defined a Groovlet named showGoals.groovy that uses the App Engine datastore low-level API to retrieve them and forward to the view goals.gtpl. In the view you can add a new goal or edit/delete existing goals.

war/WEB-INF/groovy/showGoals.groovy:

import com.google.appengine.api.datastore.Query
import com.google.appengine.api.datastore.PreparedQuery
import static com.google.appengine.api.datastore.FetchOptions.Builder.*

log.info "Getting all goals"

def query = new Query("Goal")
PreparedQuery preparedQuery = datastore.prepare(query)
query.addSort("created", Query.SortDirection.DESCENDING)
def goals = preparedQuery.asList(withDefaults())

request.goals = goals

forward '/WEB-INF/pages/goals.gtpl'

war/WEB-INF/pages/goals.gtpl:

<% include '/WEB-INF/includes/header.gtpl' %>

<h2>Goals</h2>

<a href="/goals/add">Add Goal</a>
<br><br>
<table border="1">
  <thead>
     <tr>
        <th>Name</th>
        <th>Description</th>
        <th>Created</th>
        <th>&nbsp;</th>
     </tr>
  </thead>
  <tbody>
     <% request.goals.each { goal -> %>
        <tr>
           <td>${goal.name}</td>
           <td>${goal.description}</td>
           <td>${goal.created}</td>
           <td><a href="/goals/delete/${goal.key.id}">Delete</a> | <a href="/goals/edit/${goal.key.id}">Edit</a></td>
        </tr>
     <% } %>
  </tbody>
</table>

<% include '/WEB-INF/includes/footer.gtpl' %>

The operations insert and edit share a common view named goal.gtpl. Based on whether the request contains the attribute goal the view will be in insert or edit mode. In case an existing Goal entity was provided the form gets populated with the entity data and is ready to be updated. The Goal to be edited will be retrieved by providing the entity’s ID in the editGoal.groovy Groovlet.

war/WEB-INF/groovy/editGoal.groovy:

import com.google.appengine.api.datastore.KeyFactory
import com.google.appengine.api.datastore.Key

log.info "Editing goal"

def id = Long.parseLong(params.id)
Key key = KeyFactory.createKey("Goal", id)
def goal = datastore.get(key)

request.goal = goal
forward '/WEB-INF/pages/goal.gtpl'

war/WEB-INF/pages/goal.gtpl:

<% include '/WEB-INF/includes/header.gtpl' %>

<%
  def goal = request.getAttribute("goal")
  boolean existingKey = goal?.key
  String action = !existingKey ? 'Add' : 'Update'
%>

<h2>${action} Goal</h2>

<form action="/goals/${!existingKey ? 'insert' : 'update'}" method="POST">
   <table border="0">
      <tbody>
         <tr>
            <td>Name:</td>
            <td><input type="text" name="name" value="${goal?.name ? goal.name : ''}"></td>
         </tr>
         <tr>
            <td>Description:</td>
            <td><input type="text" name="description" value="${goal?.description ? goal.description : ''}"></td>
         </tr>
      </tbody>
      <% if(existingKey) { %>
         <input type="hidden" name="id" value="${goal.key.id}">
      <% } %>
   </table>
   <br>
   <input type="submit" value="${action}">
   <input type="button" value="Cancel" onclick="javascript:document.location.href = '/goals';">
</form>

<% include '/WEB-INF/includes/footer.gtpl' %>

At last we need to implement the Groovlets for persisting the changes to your entity. All of them are quite straight forward to implement. Gaelyk’s abstractions for the datastore work quite nicely.

war/WEB-INF/groovy/insertGoal.groovy:

import com.google.appengine.api.datastore.Entity

log.info "Inserting goal"

def goal = new Entity("Goal")
goal.name = params.name
goal.description = params.description
goal.created = new Date()
goal.save()

redirect '/goals'

war/WEB-INF/groovy/updateGoal.groovy:

import com.google.appengine.api.datastore.KeyFactory
import com.google.appengine.api.datastore.Key

log.info "Updating goal"

def id = Long.parseLong(params.id)
Key key = KeyFactory.createKey("Goal", id)
def goal = datastore.get(key)
goal.name = params.name
goal.description = params.description
goal.save()

redirect '/goals'

war/WEB-INF/groovy/deleteGoal.groovy:

import com.google.appengine.api.datastore.KeyFactory
import com.google.appengine.api.datastore.Key

log.info "Deleting goal"

def id = Long.parseLong(params.id)
Key key = KeyFactory.createKey("Goal", id)
key.delete()

redirect '/goals'
Advertisements

Grails on Google App Engine – Part 3: Persistence

October 17, 2010 at 5:10 pm | Posted in GAE, Grails, Groovy, Java, Web | 4 Comments
Tags: , , , , , , , ,

The App Engine plugin installation script will prompt you for the persistence option you want to choose in your project. Google lets you select only one of its persistence implementations for BigTable: JDO or JPA. Using plain GORM is not an option. I decided to go with JDO for the simple fact that there’s more documentation for JDO than for JPA. Despite the fact that there’s documentation it certainly doesn’t cover all the questions I had. I found myself digging for answers to undescriptive exception messages for hours and hours in forums. Implementing persistence for my relatively easy domain model by far took the most of my development time making it a very frustrating experience. There are some good blogs about JDO persistence in App Engine that might be helpful to you: Google App Engine Java Persistence, Google App Engine Java Experiments. Often times I could only make things work by trail and error especially when it comes to modeling relationships.

Domain Classes

Grails domain classes have to reside in grails-app/domain and must be placed in a package. Only then the DataNucleus tool will enhance your classes. App Engine introduces its own data types. You definitely want to have a look at the list before you define the data types to your fields. It’s really a hassle to change the data type after you deployed your application to production and want to change the data type. Usually that means that you have to run a data migration script because App Engine might throw an exception on you for existing data. In my application I used the data type com.google.appengine.api.datastore.Key as primary key. The class Key has a String representation that you can use if you want to send it to your client code. This String representation can be converted back to a Key. The class you use for that is com.google.appengine.api.datastore.KeyFactory.

String bookmarkId = KeyFactory.keyToString(bookmark.key)
Key key = KeyFactory.stringToKey(bookmarkId)

You can try to exclude the default domain object fields id and version. However, that doesn’t quite propagate down to App Engine. The DataStore Viewer still shows both fields. I am not sure if there’s a different way around it.

static mapping = {
   id visible: false
   version visible: false
}

Defining relationships between domain objects can be tedious. More than often you run into funny error messages trying to store or retrieve data. You got to read the App Engine manual very closely to make them work. Especially when it comes down to defining many to many and self-referential owned relationship it can get hairy. Often times I had to fall back to just reference the Key of a different domain class to create a relationship.

Service Classes

The App Engine plugin automatically configures and provides you with Spring’s JdoTemplate that can be injected into your Grails services. Transaction boundaries can be set by wrapping your database interactions into a transactionTemplate. The JdoTemplate is easy to use and shouldn’t be new to you if you used one of Spring’s template implementations before like the JdbcTemplate. In some cases I had to fall back to the JDO low-level API because the JdoTemplate doesn’t provide the functionality. For my needs I created a super class that injects the JdoTemplate and the transactionTemplate.

class JdoService {
   def jdoTemplate
   def transactionTemplate
}

By default the JDO persistence configuration (jdoconfig.xml) created by the App Engine plugin auto-creates transactions. If you want to have full control over your transactions use the transactionTemplate. When you want to operate on more than one domain object within a transaction you have to set the property datanucleus.appengine.autoCreateDatastoreTxns to false.

<?xml version="1.0" encoding="utf-8"?>
<jdoconfig xmlns="http://java.sun.com/xml/ns/jdo/jdoconfig"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://java.sun.com/xml/ns/jdo/jdoconfig">

    <persistence-manager-factory name="transactions-optional">
        ...
        <property name="datanucleus.appengine.autoCreateDatastoreTxns" value="false"/>
    </persistence-manager-factory>
</jdoconfig>

Using the JdoTemplate doesn’t require you to close your connection. This is being handled by Spring. This is an example on how I used it:

def findAllByUser() {
   def bookmarks = []

   transactionTemplate.execute( { status ->
      def user = getLoggedInUser()
      bookmarks = jdoTemplate.find(Bookmark, "userKey == userKeyParam", "com.google.appengine.api.datastore.Key userKeyParam", [user.key] as Object[], "created asc")
   } as TransactionCallback)

   bookmarks
}

Despite the benefits of JdoTemplate you sometimes have to leverage the low-level API. However, you don’t have to create a class to retrieve the PersistenceManagerFactory. You can easily get it from the JdoTemplate. Make sure you close your resources after you used them. I wrote a little helper class to does that. One of the reasons I had to use the low-level API was to implement pagination. I chose to go down the naive, classic path of counting all records of a specific domain object and then retrieve the a list of these objects for the page we want to display. Using the range operator has implications for performance: the datastore must retrieve and then discard all results prior to the starting offset. For example, a query with a range of 5, 10 fetches 10 results from the datastore, then discards the first 5 and returns the remaining 5 to the application. I don’t expect users to have thousands and thousands of bookmarks in one folder so this would be a minor issue. Furthermore, I wanted to get up and running as soon as possible. Alternatively, this posting describes a performant but harder to implement solution.

def findAllByUserInRange(offset, max) {
   def bookmarks = []
   def total

   transactionTemplate.execute( { status ->
      def user = getLoggedInUser()
      Query query

      try {
         final filter = "userKey == userKeyParam"
         final parameters = "com.google.appengine.api.datastore.Key userKeyParam" 
         query = jdoTemplate.getPersistenceManagerFactory().getPersistenceManager().newQuery(Bookmark)
         query.setFilter(filter)
         query.declareParameters(parameters);
         query.setResult("count(this)")
         total = query.execute(user.key)
         query = jdoTemplate.getPersistenceManagerFactory().getPersistenceManager().newQuery(Bookmark)
         query.setFilter(filter)
         query.declareParameters(parameters);
         query.setOrdering("created asc")
         query.setRange(offset, offset + max)
         bookmarks = query.execute(user.key)
      }
      finally {
         JdoUtil.closeAll(query)
      }
   } as TransactionCallback)

   new Page(offset, max, total, bookmarks)
}

Indexes

The App Engine datastore maintains an index for every query an application intends to make. As the application makes changes to datastore entities, the datastore updates the indexes with the correct results. When the application executes a query, the datastore fetches the results directly from the corresponding index. As this might hold true for your development machine this isn’t necessarily true when you deploy your application to App Engine. Often times App Engine didn’t indicate that I have a missing index. You either have to analyze your queries very closely and add the indexes manually in grails-app/conf/datastore-indexes.xml or you’ll find yourself seeing exceptions. These exception can be fixed easily. However, you have to redeploy your application. It might take App Engine a while (sometimes 30 minutes or longer) to build the indexes once you have them defined. Indexes that are in the process of being built cannot serve the query causing an exception. This makes it impossible to deploy a new index without problems for your users. At the moment there’s no way to delete indexes over the administration console.

More on this topic:

Grails on Google App Engine – Part 2: Project Setup

October 17, 2010 at 5:09 pm | Posted in GAE, Grails, Groovy, Java, Web | 4 Comments
Tags: , , , , , , , ,

Setting up Grails for App Engine on my favorite IDE IntelliJ was as easy as installing the App Engine plugin. At the time of writing the latest plugin version is 0.8.10. It is by far not free of bugs but gets you where you want. It provides you with the basic setup to build, run and deploy your project in your development environment. The plugin was fully compatible with Grails 1.3.1. I didn’t try out to upgrade my project to the latest version of Grails yet. Stick to the plugin documentation to get the basic setup going. To be able to reference the GAE/J classes I dropped in the JAR files appengine-api-1.0-sdk-1.3.7.jar and jdo2-api-2.3-eb.jar into my project’s lib directory. It was extremely helpful to set up Log4J correctly right away. Earlier versions of the plugin had some Log4J-related bugs but they seem to be fixed now. Here’s an excerpt of my Config.groovy:

log4j = {
   appenders {
      console name:'stdout', layout:pattern(conversionPattern: '%d{dd MMM yyyy HH:mm:ss,SSS} [%t] %-5p %c{2} - %m%n')
   }

   debug  'grails.app', 'com.favalike'
   ...
}

The plugin still has a bug related to i18n. In case you’re using one of the messages.properties files to store internalized texts that would leave these files empty after building the WAR file. I could still make it work by applying the workaround described in the JIRA ticket. You got to change your Bootstrap.groovy as follows and build your WAR file for production environment: grails -Dgrails.env=production app-engine package.

import grails.util.Environment

class BootStrap {
   def messageSource

   def init = { servletContext ->
      if(Environment.getCurrent() == Environment.PRODUCTION) {
         messageSource.basenames = ['WEB-INF/grails-app/i18n/messages']
         messageSource.clearCache()
      }
   }

   def destroy = {
   }
} 

To improve performance from the get go I made sure I had precompilation enabled. Furthermore, static files can get served by dedicated servers and caches. Both settings are made in appengine-web.xml.

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
   ...
   <precompilation-enabled>true</precompilation-enabled>
   <static-files>
      <include path="/favicon.ico" />
      <include path="/css/**.css" />
      <include path="/images/**.*" />
      <include path="/js/**.js" expiration="1d" />    
   </static-files>
</appengine-web-app>

More on this topic:

Grails on Google App Engine – Part 1: A Case Study

October 17, 2010 at 5:08 pm | Posted in GAE, Grails, Groovy, Java, Web | 5 Comments
Tags: , , , , , , , , ,

Soon after Google had announced App Engine for Java I started to play around with the platform. Despite the limitations that App Engine imposes on developers I really wanted to know what all the buzz is about. Not only did I want to deploy a useful, non-trivial application I also wanted to get a feel for App Engine’s production readiness. Most of today’s developers are embracing (web) frameworks that help them to focus on implementing business logic. This might become an issue with App Engine for multiple reasons. Adding Megabytes of JAR files to your application increases the cold start up time. Furthermore, App Engine restricts the usage of certain specifications, technologies and frameworks. Over the past two years I became a big fan of Groovy and Grails. Therefore, I wanted to leverage the ease of use of Grails to implement a solution on App Engine. All I needed to get started was the Grails App Engine plugin. Even though I read about Gaelyk I wasn’t fully convinced that it would be able to provide me with all the functionality I needed to implement my application e.g. no support for taglibs or Sitemesh.

FavalikeTo limit the amount of work I’d have to put into this experiment I chose to implement a online bookmark manager with limited functionality. The features I wanted to cover should also take advantage of some of the services provided by App Engine. My plan was to use the Mail service and the URL Fetch API to get a feel on how easy it is to use them.

  • User registration and confirmation via email
  • User authenication
  • Management of bookmarks in folders per user
  • Tagging of bookmarks
  • Marking bookmarks as favorites
  • Automatic retrieval of URL meta data
  • Export of bookmarks as XML and HTML

The final result became Favalike which uses Grails 1.3.1 and the App Engine plugin 0.8.10. The following postings will describe my experience and the challenges I had to face in detail.

More on this topic:

Create a free website or blog at WordPress.com.
Entries and comments feeds.