Inefficient Session Implementation on Google App Engine / Java & GAE Critics

Almost two months ago I released my first Google App Engine (GAE) application called www.krisentalk.de.
In general it is best practice to do some profiling and check for bottlenecks before going to production. So did I.

GAE offers a nice tool called “Appstats” which profiles current requests.
Studying the profiling results I had an unpleasant surprise. From time to time one request contained around 40 RPC (RPC=Remote Procedure Calls=Remote GAE Services). That was strange, because I never implemented more than 4 RPC per request. The following screenshot shows this:



What was happening?

It is noticeable that there are many pairs of memcache/datastore-Gets. It turns out that each pair is an attempt to read the session-state. But why there are for the same page sometimes just one session-read and sometime around 40 (like in the screen above)?

A little more digging has shown what’s going on: Whenever there is an invalid session there are many unsuccessful readings for session-state. In other words: The Browser sends a JSESSION cookie, but the session on the GAE is already invalidated (e.g. session got a timeout). My application is using Spring MVC and Freemarker. These frameworks (as many others) try to read the session more than once per request.

First Conclusion

Google App Engine has a grossly inefficient implementation for reading (invalided) session state. Whenever a session is invalid, multiple times RPCs are used just for checking again and again that there is no valid session. There shouldn’t be more than one attempt to read the session via Remote-Calls per request.
I issued a ticket on GAE for this: #7355

Workaround

As a workaround I implemented a request interceptor that checks if a user sends a JSESSION cookie that refers to an invalidated session. If this is the case the user’s JSESSION cookie is removed. Hence, the follow-up requests of this user don’t lead to the unnecessary RP-Calls anymore. The source in a Spring’s HandlerInterceptorAdapte looks like this:

 
void post(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
  Cookie cookie = WebUtils.getCookie(request, WebUtils.COOKIE_SESSION_NAME);
  if ( cookie != null  && request.getSession(false) == null) {                        
    MyWebUtils.deleteCookie("JSESSIONID", request, response);
  }
}

General GAE critics

One of my (and I think most other GAE users) biggest critics is that Google let their customers pay if they have an inefficiency or failure on their GAE platform. You can imagine what GAE’s bad session-implementation shown above would cost a customer with a high traffic application! On the GAE-user-group there was a nice joke that summarizes this to the point:

thecheatah: Why are my requests taking > 1 second all of a sudden?
Jeff: It's the end of the financial quarter and GAE needed to make revenue targets ;-)

The second (even worse) complain is the very bad user support/communication of Google regarding GAE’s customer. Google seems to have a politics of “ignore customer complains”. If you have problems like your costs go up or your application fails because of GAE’s reliability problems you are lost!
A quote of somebody that summarizes the feeling of affected GAE users:

“Some attention from google after a long time is nice to see. but still not clear when this issue get resolved.
More importantly concerns regarding credibility still in the air. I think many including us are planning to get out of GAE. Not because of the issue but the way such a critical issue is being handeled.”

Check the following issue for the quote above and for a serious reliability issue that Google did not response for more than one month.
http://code.google.com/p/googleappengine/issues/detail?id=7133

Comparing this to other platforms like Grails (from Spring Source) there are two totally separate worlds regarding “community-treatment”.

Google App Engine & Two new Babies

My gosh! Many activities were going on the last months. First our lovely new son arrived:



Then I finally released a new website called www.krisentalk.de. It’s all about the financial crisis in Europe and the rest of the world (called "Krise" in German). The content is community-driven.



I developed krisentalk.de on the Google App Engine (GAE) / Java. Starting with a little evaluation on cloud-platforms, I decided to go with GAE.
All in all it was a fun and expedient experience. Nevertheless there are pitfalls and Google’s cloud-platform still feels “green” on some edges. I will blog about some of the pitfalls later on.

I chose the following frameworks, which work well for me.

• GAE Java Runtime
• Spring 3.0.x (particularly Spring MVC)
• Freemarker
• Sitemesh
• GAE’s Java Datastore API (wrapped with the good old Data Access Objects [DAO pattern])

Grails wasn’t an option because it doesn’t work very well on GAE.

My positive résumé on Google App Engine:

• Setup, deployment and scaling is a no-brainer
• The specific GAE API is proper and easy to use
• Local development environment based on Eclipse works fine
• Out of the box administration tools and monitoring capabilities available
• Common services like emailing, asynchronous queuing, mem-caching, authentication etc. are available and easy to use
• Scaling (as far as my JMeter tests predict)

My negative résumé on Google App Engine:

• Some restrictions on the usage of Java’s SDK API (and thus some frameworks are not working well with GAE)
• Feedback and support from Google and the community is improvable compared to others (e.g. Grails)
• Critical bugs take sometimes too long to get recognized and fixed from Google (e.g. Website broke because wrong CSS mime-type)
• Small down-times from 2 to 5 minutes in 24h the first days (running on HR Datastore)
• Whenever there is a misbehavior in Google’s resource/scaling management, the customer has to pay. Nice business model: Introduce a misbehavior on customer’s platform and revenue goes up. ;) (e.g. Scheduler behavior and Slow requests). I will blog another example the next days.

Grails Logging & New Website launched

This month I’ve got special news. We just launched our new family site http://scheelethek.de. :)
It’s developed in Grails and shows once more that Grails is the right application framework for fast, fun and reliable web-development.
Ok, it took some time to launch it. But that wasn’t Grails’ fault. There was just no time besides my professional and family life. I coded up to one hour every several weeks only.
But Grails makes it easy to stay connected without the need to read up on the previous development again and again.

This month I decided to post the logging configuration of this application. This is a part of Grails that isn’t very intuitive.
I think the logging requirements of my application are sophisticated but still very common:

• In Development and Test-Mode logging should be done to the console (“Stdout”)
• In Production-Mode logging should go to a file (and “Stdout” should be deactivated)
• The file should be recreated on size-threshold or time (“rollingFile”)
• In Development and Test-Mode logging should be done on “Debug-Level”; for Production it should be the “Info-Level”
• The Logging should include the name of the logged in user (or session-id if not logged in). The format is as follows:
  2011-09-11 17:19:02,290 [http-8080-5] (0646291FBCC5F8E95BE6E10C4AB9DFBE) INFO user.RegistrationController - User 'Hans Wurst' has registered. Sending email to: hans_wurst@sdfds.de
• When Warnings or Errors are logged, an eMail should be sent directly into my mailbox; including the message
• Of course: All components of my application should have logging activated

My configuration for Grails 1.3.7 in the Config.groovy looks like this:

def logPattern = '%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] (%X{user}) %p %c{2} - %m%n'
log4j = {
    
	appenders {
   	 
      	console name:'stdout', layout:pattern(conversionPattern: logPattern)
      	'null' name:'stacktrace'
     	 
      	environments {
          	production {
               // rolling file per size
              	rollingFile name:'fileAppender', maxFileSize:'2MB', file: "/opt/mypath/myLog.log", layout:pattern(conversionPattern: logPattern)
                // rolling file per date
                //appender new DailyRollingFileAppender(name: 'file', datePattern: "'.'yyyy-MM-dd", fileName: "/opt/mypath/myLog.log",layout: pattern(conversionPattern: logPattern))
              	'null' name:'stdout'
              	// mail appender
              	appender new org.apache.log4j.net.SMTPAppender(
                  	name: 'mail',
                  	to: 'myemail@mysdfwhatever.de',
                  	from: 'noreply@mysdfwhatever.de',
                  	threshold: org.apache.log4j.Level.WARN,
                  	bufferSize: 1,
                  	subject: "Error on ScheeleThek",
                  	layout: pattern(conversionPattern: logPattern),
                  	//SMTPDebug: true,
                  	SMTPHost: 'localhost'
              	)
          	}
      	}
	}
    
	root {
    	     error 'stdout', 'fileAppender', 'mail'
	}

	error  'org.codehaus.groovy.grails.web.servlet',   
       	'org.codehaus.groovy.grails.web.pages',  
       	'org.codehaus.groovy.grails.web.sitemesh',  
       	'org.codehaus.groovy.grails.web.mapping.filter',  
       	'org.codehaus.groovy.grails.web.mapping',  
       	'org.codehaus.groovy.grails.commons',  
       	'org.codehaus.groovy.grails.plugins',  
       	'org.codehaus.groovy.grails.orm.hibernate',  
       	'org.springframework',
       	'org.hibernate',
       	'net.sf.ehcache.hibernate'

	warn   'org.mortbay.log'
    
	debug  'de.scheelethek',
        	'grails.app'
	 
	environments {
   	   production {
        	  info	'de.scheelethek',
        	    	'grails.app'
   	    }
	}
  	 
}

As described above the logging outputs the name or session-id of the current user. This is very useful when all activities of a user should be collected later on. To make this work you need a Grails filter like this:

      class UtilityFilters {
    
	SpringSecurityService springSecurityService
    
	def filters = {
   	 
   	 
    	/**
     	* Used for setting the log4j diagnostic context:  Username if available; otherwise the session-id
     	*/
    	logContext (controller: "*", action: "*") {
       	 
        	before = {
            	if (springSecurityService.isLoggedIn()) {
                	MDC.put('user',"${springSecurityService.principal?.username}")
            	} else {
                	if (RequestContextHolder.getRequestAttributes()!=null) {   	// if we have a valid web-request
                    	MDC.put('user',"${session.id}")
                	}
            	}
        	}
       	 
        	afterView = {
            	MDC.remove 'user'
         	}
       	 
    	}
   	 
	}

}

Grails Security - XSS Prevention using Html-Codecs

There are so many applications out there that don’t care about Cross-site scripting (XSS) attacks. Try some of your favourite sites. Just type something like

<script type="text/javascript">alert("attack")</script>

into an input field and check if you get such a message:



Grails has a nice feature for fixing this basic XSS issue.
All you need to do is set the default-codec to html in your Config.groovy:

grails.views.default.codec = "html"

Now every expression inside the ${...} syntax gets encoded in your GSPs:

${'<script type="text/javascript">alert("attack")</script>'}

would become

<script type="text/javascript">alert("attack")</script>

Beyond that you need to be careful with Tags (e.g. <g:message...>). Output from those Tags are not encoded by the default-codec. It’s up to the Tag’s implementation.

But what’s with the exception of the rule? What if you have some Html output which should not be encoded, but printed like it is? Or what if you have an email-template which is sent as plain text where variables are not allowed to be encoded?
You can do the following to avoid encoding in special cases :

• Use the syntax <%=expression%> (instead of ${expression})
• Put <%@page defaultCodec="none"%> on the top of your template. All ${expression} inside this GSP are not encoded

There is still one small criticism regarding the default codec in Grails:

If you generate a new Grails project the default codec is ‘none’. I saw so many projects in my Grails consulting that didn’t change this (because they did not know or just forgot it). To change this afterwards is a pain and leads to increased effort.
I hope in Grails Version 2.0 the default-codec will change to ‘html’.

Grails - Accessing the Http-Session from Services

Holy Cow. I didn’t make it the last six months to write a post here. Too much stuff is going on in private and business life. But I promise improvement by writing an article at least once a month.
As I’m currently working most of the time on Grails projects, I restart the Blog with a little Grails series. Starting with the question: Is it useful to access a http-session in a Grails Service.

In general it isn't a very good idea to access the web session from within Services.
First of all you never know if the web session is available: Think about calling the service method from a job or JMS call (instead of a controller). There wouldn't be any http-request associated with the current thread.
Second, it's a common and good design pattern to separate the service logic from client specific stuff.

But there are circumstances where it may be adequate to access the session within a service.
One way to do it would be to use a method parameter for transferring the session object.
However, that could clutter method signatures which isn't a nice design also.
Another way could be to use Spring's session scope beans. This seems to be right for
many cases, but could be a little too much overhead for simple use cases.

A pretty alternative is to implement a small service that encapsulates the access to
the http-session in a safe manner.

This service could look like this:

 class WebUtilService {

    void withSession (Closure closure) {
        try {
            GrailsWebRequest request = RequestContextHolder.currentRequestAttributes()
            GrailsHttpSession session = request.session
            closure.call(session)
        }
        catch (IllegalStateException ise) {
            log.warn ("No WebRequest available!")
        }
    }
    
}

It's used like this:

class MyService {
    
    WebUtilService webUtilService
    
    void doSomething() {
        webUtilService.withSession { HttpSession session ->
            log.info ( session.myValue )
            session.newValue = 'Possible, but should be exceptional'
        }
    }

}