Wednesday, November 15, 2017

Blog switched to Jobpushy

The last 18 months I spent a lot of my time to build and promote the new service Jobpushy.  It's all about finding the right IT-Jobs.  We wanna make software-developer, administrators and designer happy in their jobs.

Our service is based on a sophisticated software using technologies around fulltext-search, Web-crawling, maschine-learning, microservices in the cloud.  Tools like Elasticsearch, Grails/Groovy and Java are used and deployed in the AWS Cloud.

On the new blog I write from time to time also about technical-topics. It can be found here:

Jobpushy-Blog        (Most articles are written in German currently)

CU on Jobpushy

Friday, January 11, 2013

Grails versus Google App Engine (GAE)

Currently I’m involved in implementing the promising web-service called As usual it starts with the question which platform would be best. I did a lot of projects and consulting in the area of the Grails-Framework and around the Google App Engine (GAE). Both have strengths and both can be a perfect match for developing fast a new software-startup.

Why “versus” and not “with”

Some people may remark that Grails can run on top of GAE.  That’s true. In theory you can grab the GAE Grails-Plugin and would be able to deploy your Grails application to the Google App Engine. But you shouldn’t do this. The top three reasons “why not to use Grails on Google App Engine”:

  • The GAE Plugin of Grails is maintained badly. It breaks regularly with every new Grails-Version.
  • Grails’ functionalities are restricted on GAE.   (Because of the non-relational database and the Java API restrictions on Google App Engine)
  • Startup Time of Grails Applications is unacceptable slow on GAE.  This leads to many downtimes for your users.

Google App Engine Trade-Offs

GAE is a PaaS  (Platform as a Service)  that supports Java and has its strength in big data and scaling.  It works well with Java applications that have a lean data-model and not a lot of dependencies inside application’s modules. For applications with a huge amount of data, demanding processing work and the requirement to scale up fast,  Google App Engine is especially productive.

Please note the top three reasons “when Google App Engine should not be used”:

  • You are going to implement traditional application with large (relation) data model, use of transactions and many functional modules.
    (Examples would be enterprise-applications or complex user-communities.  Implementing those on GAE would be a pain.)
  • You would like to use your favorite frameworks like Spring or Hibernate and wouldn’t be locked into Google App Engine at any time.
    (Traditional frameworks like Spring don’t work well with GAE. The main problem is much increased startup time that leads to downtimes.)
  • You would like to have feedback and support from your cloud-provider if the platform has issues, payments did not succeed or other mystery stuff going on.
    (Google sucks in customer support!)

Grails – the swiss army knife

Most of my current projects I’m doing with Grails. It’s a great framework for fast development of web based applications. It’s built on the foundation of the established Spring- and Hibernate-Frameworks, adds Groovy to allow dynamic programming and offers design patterns that leads to efficient and DRY source-code.

In general I recommend giving Grails a try.  Nevertheless, here are the top three reasons “when not to use Grails”:

  • You know for sure that your application will get traffic as hell and you need fast and cheap scaling. 
    (It’s possible and done with Grails, but it’s much more costly than using a scalable PaaS.)
  • You, your colleagues and/or your boss don’t like dynamic programming languages like Groovy.
    (You can try to convince them (Groovy 2.x can compile static!). But there is the chance that you have to stick with the good, old and boring Java language and frameworks.)
  • You always have to use the latest and brand new framework versions. 
    (New Grails releases have often some major bugs. Hence, you should go with the “LatestVersionNumber - 0.1” version.)

Conclusion – What’s with Linkanu

Now, on which platform will be implemented?  I chose Google App Engine for this.  Linkanu is a searchengine-type application. High-level it works like this:

The user enters two personal input values.  The result would be a couple of websites and web-offerings that the user always was looking for.  It is based on a special “comparing-algorithm” and a huge index of data.  Linkanu is one of the exceptions which proves my rule “in general go with Grails, unless your application has special requirements”. Check it out:

Monday, April 16, 2012

Inefficient Session Implementation on Google App Engine / Java & GAE Critics

Almost two months ago I released my first Google App Engine (GAE) application called
In general it is best practice to do some profiling and check for bottlenecks before going to production. So did I.

GAE offers a nice tool called “Appstats” which profiles current requests.
Studying the profiling results I had an unpleasant surprise. From time to time one request contained around 40 RPC (RPC=Remote Procedure Calls=Remote GAE Services). That was strange, because I never implemented more than 4 RPC per request. The following screenshot shows this:

What was happening?

It is noticeable that there are many pairs of memcache/datastore-Gets. It turns out that each pair is an attempt to read the session-state. But why there are for the same page sometimes just one session-read and sometime around 40 (like in the screen above)?

A little more digging has shown what’s going on: Whenever there is an invalid session there are many unsuccessful readings for session-state. In other words: The Browser sends a JSESSION cookie, but the session on the GAE is already invalidated (e.g. session got a timeout). My application is using Spring MVC and Freemarker. These frameworks (as many others) try to read the session more than once per request.

First Conclusion

Google App Engine has a grossly inefficient implementation for reading (invalided) session state. Whenever a session is invalid, multiple times RPCs are used just for checking again and again that there is no valid session. There shouldn’t be more than one attempt to read the session via Remote-Calls per request.
I issued a ticket on GAE for this: #7355


As a workaround I implemented a request interceptor that checks if a user sends a JSESSION cookie that refers to an invalidated session. If this is the case the user’s JSESSION cookie is removed. Hence, the follow-up requests of this user don’t lead to the unnecessary RP-Calls anymore. The source in a Spring’s HandlerInterceptorAdapte looks like this:

void post(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
  Cookie cookie = WebUtils.getCookie(request, WebUtils.COOKIE_SESSION_NAME);
  if ( cookie != null  && request.getSession(false) == null) {                        
    MyWebUtils.deleteCookie("JSESSIONID", request, response);

General GAE critics

One of my (and I think most other GAE users) biggest critics is that Google let their customers pay if they have an inefficiency or failure on their GAE platform. You can imagine what GAE’s bad session-implementation shown above would cost a customer with a high traffic application! On the GAE-user-group there was a nice joke that summarizes this to the point:

thecheatah: Why are my requests taking > 1 second all of a sudden?
Jeff: It's the end of the financial quarter and GAE needed to make revenue targets ;-)

The second (even worse) complain is the very bad user support/communication of Google regarding GAE’s customer. Google seems to have a politics of “ignore customer complains”. If you have problems like your costs go up or your application fails because of GAE’s reliability problems you are lost!
A quote of somebody that summarizes the feeling of affected GAE users:

“Some attention from google after a long time is nice to see. but still not clear when this issue get resolved.
More importantly concerns regarding credibility still in the air. I think many including us are planning to get out of GAE. Not because of the issue but the way such a critical issue is being handeled.”

Check the following issue for the quote above and for a serious reliability issue that Google did not response for more than one month.

Comparing this to other platforms like Grails (from Spring Source) there are two totally separate worlds regarding “community-treatment”.

Monday, March 5, 2012

Google App Engine & Two new Babies

My gosh! Many activities were going on the last months. First our lovely new son arrived:

Then I finally released a new website called It’s all about the financial crisis in Europe and the rest of the world (called "Krise" in German). The content is community-driven.

I developed on the Google App Engine (GAE) / Java. Starting with a little evaluation on cloud-platforms, I decided to go with GAE.
All in all it was a fun and expedient experience. Nevertheless there are pitfalls and Google’s cloud-platform still feels “green” on some edges. I will blog about some of the pitfalls later on.

I chose the following frameworks, which work well for me.

  • GAE Java Runtime
  • Spring 3.0.x (particularly Spring MVC)
  • Freemarker
  • Sitemesh
  • GAE’s Java Datastore API (wrapped with the good old Data Access Objects [DAO pattern])

Grails wasn’t an option because it doesn’t work very well on GAE.

My positive résumé on Google App Engine:

  • Setup, deployment and scaling is a no-brainer
  • The specific GAE API is proper and easy to use
  • Local development environment based on Eclipse works fine
  • Out of the box administration tools and monitoring capabilities available
  • Common services like emailing, asynchronous queuing, mem-caching, authentication etc. are available and easy to use
  • Scaling (as far as my JMeter tests predict)

My negative résumé on Google App Engine:

  • Some restrictions on the usage of Java’s SDK API (and thus some frameworks are not working well with GAE)
  • Feedback and support from Google and the community is improvable compared to others (e.g. Grails)
  • Critical bugs take sometimes too long to get recognized and fixed from Google (e.g. Website broke because wrong CSS mime-type)
  • Small down-times from 2 to 5 minutes in 24h the first days (running on HR Datastore)
  • Whenever there is a misbehavior in Google’s resource/scaling management, the customer has to pay. Nice business model: Introduce a misbehavior on customer’s platform and revenue goes up. ;) (e.g. Scheduler behavior and Slow requests). I will blog another example the next days.

Tuesday, September 13, 2011

Grails Logging & New Website launched

This month I’ve got special news. We just launched our new family site :)
It’s developed in Grails and shows once more that Grails is the right application framework for fast, fun and reliable web-development.
Ok, it took some time to launch it. But that wasn’t Grails’ fault. There was just no time besides my professional and family life. I coded up to one hour every several weeks only.
But Grails makes it easy to stay connected without the need to read up on the previous development again and again.

This month I decided to post the logging configuration of this application. This is a part of Grails that isn’t very intuitive.
I think the logging requirements of my application are sophisticated but still very common:

  • In Development and Test-Mode logging should be done to the console (“Stdout”)
  • In Production-Mode logging should go to a file (and “Stdout” should be deactivated)
  • The file should be recreated on size-threshold or time (“rollingFile”)
  • In Development and Test-Mode logging should be done on “Debug-Level”; for Production it should be the “Info-Level
  • The Logging should include the name of the logged in user (or session-id if not logged in). The format is as follows:
    2011-09-11 17:19:02,290 [http-8080-5] (0646291FBCC5F8E95BE6E10C4AB9DFBE) INFO user.RegistrationController - User 'Hans Wurst' has registered. Sending email to:
  • When Warnings or Errors are logged, an eMail should be sent directly into my mailbox; including the message
  • Of course: All components of my application should have logging activated

My configuration for Grails 1.3.7 in the Config.groovy looks like this:
def logPattern = '%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] (%X{user}) %p %c{2} - %m%n'
log4j = {
	appenders {
      	console name:'stdout', layout:pattern(conversionPattern: logPattern)
      	'null' name:'stacktrace'
      	environments {
          	production {
               // rolling file per size
              	rollingFile name:'fileAppender', maxFileSize:'2MB', file: "/opt/mypath/myLog.log", layout:pattern(conversionPattern: logPattern)
                // rolling file per date
                //appender new DailyRollingFileAppender(name: 'file', datePattern: "'.'yyyy-MM-dd", fileName: "/opt/mypath/myLog.log",layout: pattern(conversionPattern: logPattern))
              	'null' name:'stdout'
              	// mail appender
              	appender new
                  	name: 'mail',
                  	to: '',
                  	from: '',
                  	threshold: org.apache.log4j.Level.WARN,
                  	bufferSize: 1,
                  	subject: "Error on ScheeleThek",
                  	layout: pattern(conversionPattern: logPattern),
                  	//SMTPDebug: true,
                  	SMTPHost: 'localhost'
	root {
    	     error 'stdout', 'fileAppender', 'mail'

	error  'org.codehaus.groovy.grails.web.servlet',   

	warn   'org.mortbay.log'
	debug  'de.scheelethek',
	environments {
   	   production {
        	  info	'de.scheelethek',
As described above the logging outputs the name or session-id of the current user. This is very useful when all activities of a user should be collected later on. To make this work you need a Grails filter like this:
      class UtilityFilters {
	SpringSecurityService springSecurityService
	def filters = {
     	* Used for setting the log4j diagnostic context:  Username if available; otherwise the session-id
    	logContext (controller: "*", action: "*") {
        	before = {
            	if (springSecurityService.isLoggedIn()) {
            	} else {
                	if (RequestContextHolder.getRequestAttributes()!=null) {   	// if we have a valid web-request
        	afterView = {
            	MDC.remove 'user'