Google App Engine application is temporarily over its serving quota

After developing a google app engine application for two months and investing a lot of time in it, and now serving more than 500,000 page views per month I got the following error appear on my website:

"Over Quota

This Google App Engine application is temporarily over its serving quota. Please try again later."


It seems that i had a small mistake which caused runtime exceptions in some of the requests, This small mistake cause my web site to shutdown for several hours!!!!

I really don't understand Google making this error apeear, I understand that they want to secure their servers but why to shutdown an application for several hours, once the error was fixed the server should be activate again.

Now that the server is done, and many users see the error they won't use the application again!

For this reason I really don't recomment you use Google App Engine.

Google App Engine

One month after I started writing my first Google App Application I had reached more then 1000 Page View per one day!!!



One of the great advanatges of GAE over creating my own Web Server is that robutness of Google servers, My application is up 100% of the time, And after writing the application in correct way without consuming too much CPU per request and handling exceptions the application can be used anytime by the users.



The question is what would happen when GAE would support also Java, I realy don't know how I would be able to convert my application from Python to Java

Google App Engine Tips

1. Often your application would work just fine locally but after deploying it the application won't work, a reason for that is that when running locally there is no CPU limit but on Google servers there is a CPU limit, My advice is to deploy your application at night this would enable you to test your application live without interfering the users
2. Check your Logs frequently to see if your application throws exceptions to the user, You should strive to a point that no error logs would be displayed at all.
3. To delete old entities you can access from "Application Overview" then choose "Data Viewer" and then select the entity click on the top check-box and then click the delete button, once there are no rows the entity would be removed.
4. Don't upload an application if you havn't fully tested it before locally.

More tips would be posted in the next posts

My First Google App Engine Application

Ok, I finally succeed writing the Beta version of my Google App Engine application, This application translate from Hebrew Language to 15 other Langauges - Engish, Russian, Japanesse, Chinesse, Korean. The application use the datastore to hold the dictionary and also it auto-correct words which are misspelled, You can check the application at this url -

http://tirgumchik.appspot.com

Deploying such application myself which would be both robutness and scalable can cost a lost of money, If buying servers and paying for high bandwith internet connection and maintaining the DB and using Load Balancer and so on, I think that it's worth it.

Google App Engine in Java Language

As A Java Developer it is very hard to use Google App Engine.
As for now Google App Engine supports only Python Programming language, Trying learning and writing Python languages can be very frustrating when you are used to the good pleasing Eclipse IDE.

Then I found this nice project:

http://code.google.com/p/java2python/

You can write you Google App Engine Application in Java Language and then just convert it to Python and upload it to Google Servers!!!

Enjoy

Google App Engine Limits

Google App Engine has many unknown limits and resterictions, here are few of them:

• Data store is limit to 500mb
• 5,000,000 monthly page views
• 3 applications per user
• Application response time to user must be short something like 3 seconds
• Bulk upload failed to load long strings, I didn't succed upload strings which are more then 40 letters long
• Unicode charcters can't store in datastore
• Bulk upload cookie expiration is 15 minutes long, If your upload takes more time then you would need to run bulk upload several times again and again
• Number of users which can user Goolge App Engine is limit for now
• It is not possible to rewrite datastore date only to add more rows using bulk upload
• Google App Engine is limit for now only to Python programming language

Nevertheless Google App Engine works and it has many advantages which are not exists elsewhere, It is ideal for developers which don't have the resource to maintain a complex website for their applications.

Google App Engine Work !!!

After hard work and many workarounds I managed to run my first Google App Engine application !!!

And it works great, The performances are amazing, I have more than 10,000 rows in my tables and selecting the rows takes few milliseconds.

I manage to find workaround for the unicode problem by not writing unicodes in the Goole App Engine datastore, I also managed to find workaround for the bulk upload problem by loading the entities one by one and not 10 entities at a time.

So regarding my last post which suggest not to use Google App Engine, I change my mind, You can use it but you would need to work very hard to make it work.

Google App Engine and GWT

First there was GWT - Google Web Toolkit a very nice framework to develop Ajax applications in Java language which then translate to JavaScript, It worked with Tomcat Web Server. Then Google decide to release Google App Engine a service which provide the ability to develop applications in Python language and deploy them on their servers for free and provide datastore which is like DB to store data.

It seems like their is no syncrhonization at all with the two groups GWT and Google App Engine, once Google goes for Java and then for Python, Is there any direction to which Google wants to go to?

In their website they claim that they choosed Pyhton because it is secure language, There is no more secure lagnuage than Java, How would GWT developers would deploy their applications on Google App Engine, would they need now to convert their code to Python? I really confused

Why not using Google App Engine

After worked several days with Google App Engine, I was realy dissapointed, It seems that since it has just released and it is in "Preview Release" it contains some bugs and since Google choose Python as the programming language it make it hard to develop for programmers which don't know well Python, Here are the problems that I come with

• Python is VERY hard language to develop applications which required Unicode, Maybe for english applications its not so bad language but when you try writing something which is not english you need to find yourself using decode, encode, unicode and other awful methos.
• It seems that urlfetch method which retrieve data from other urls doesn't support unicode
• I tried to upload my datastore to the application server, when I do it locally it works, however when I try to upload the datastore remotly on Google Servers it fails.
• Datastore also doesn't support unicode characters, everything has to be in english
• Sometimes after uploading the application to the Google Servers it says error 500, However the application works just find locally
• Python language is the only programming language I know which requires not using Tabs inside the source code and it doesn't use { and } characters to define block but rather it looks on where the line begins. There are much more advance programming languages like Java which could fit more to be used with Google App Engine

In summary I think that I would wait some time before Google App Engine would have more stable versions.

Working with Google App Engine

I just got today invitation to try Google App Engine,
It seems that I'm one of the lucky 10,000 people that got invitation to try it.

Well first I had to admit that I don't know shit about Phyton, Google App Engine is based about Phyton language, However it seems that it is not hard to learn this langauge once you know Php, Java or other programming language.

The big advantage of Google App Engine, is that you don't have to take care about all installing and deploying applications, taking care for performance or load balancing, you get all this things for Free!!!

It seems that it is pretty easy to develop applications using Google App Engine, you just need to install phyton, and Google App Engine SDK, and thats all.

Then you are your phyton application file, and you can test it localy by running GAE (Goolge App Engine) locally.

Once you finished writing you application you just upload it to the Google Server.

The biggest advanatge of GAE in my opinion it is ability to store data and retrive data from the database using GQL Google Query Language which is just like SQL but a bit different, This allow the application writing highly scalabe SQL selects.

Once I would finish writing my application I would publish it here.
Please comment if you have any questions.

Google App Engine

Google announced yesterday about their ambitious project "Google App Engine",

The idea for Google is why we should spend so much effort developing applications? Lets make developers outside Google develop applicatons for us. This is the same idea like "YouTube" the users upload the videos for Google.

Each user gets free 500MB of persistent storage and bandwidth for about 5 million page views a month.

Don't hurry to register since the project is limited to the first 10,000 developers, And there are more than 10,000 developers which already registered.

When I tried to check the App Gallery all the applications returned the following error after trying to login: "The requested page is expiered".


Google App Engine

Build Mashup Application

In this post I would explain how easily is to build mashup web applications, and I would list providers which provide services to build Mashup applications.

Mashup is a Web Application that combines data from several external sources into the application and thereby the development of such application is wasy and fast.

Example of such external sources are

• Google Maps - Provide intergrated maps inside your own application
• Thumbshots - Provide a way to put thumbnails on your web site
• YouTube, MegaVideo, DailyMotion - Integrate videos to your web site
• Adsense, Abrite - Add ads to your web site
• Outbrain - Add rating to your blog

There a lot of other external sources, like RSS feeds and others.

There are also tools which allow to build Mashup applications On-Line, This tools allows to create workflow which get sources from external resources such as in RSS format and merge this dats into one web site, The major software vendors provide such tools -

• Yahoo Pipes
• Microsoft Popfly
• Google Mashup Editor (Beta Version)

The disadvantage of mashup applications are

• They depend on many sources, so if one of this resources is unavailable then all the mashup application becomes unavailable.
• The performance of such applications is not good since they depend on several outer resources from different locations.
• If one of the resources change it's API or start to charge for the service your Mashup application would stop working.

Cache Manager

Actually I took the following Cartoon idea from a real project which I was involved in it. To improve the performance of our application the Architect wanted us to build local cache (We couldn't use Open Source in our project).





















They gave us several months to develop it... :)

Terracotta - Simplfy Cluster Development

Cluster development is often very complicated, Many applicatoins needs to develop special infrastructure to support clustering such as Cluster cache, replication of data between the different servers in the cluster and each develop need to take into account the design and the implementation of his module to run well in cluster environement.

Terracotta comes to solve this problem, Terracota is Open Source and it is called "Network Attached Memory" or simply NAM. Terracotta take care for all the cluster issues, eliminate the developer the need to deal with all this complicated stuff.

For example java synchronized(), wait() & notify() can work exactly the same across several VMs as across threads!!!

Terracotta also provides In Memory DB which is durable to Server Crashes.

In summary if you are looking for an open source which can scale your application in little effort I recommend using Terracotta.

Execute Async Method

Sometimes you need to execute methods in async way.

There are several ways which you can do it

1. Send JMS message in J2EE Application with a Callback object, Write MDB which would listen to the Queue which you send the message to, When the MDB would finish it would call the method 'done' on the Callback if the execution succeed, and it the execution failed it would call the method 'failed' on the Callback.
2. If you write with Java 1.5 simple small Java Program and you don't need to distribute your async command in cluster environment you can use the 'Executors' class to create pool of executions in this way:

ScheduledExecutorService executorService = Executors.newScheduledThreadPool(10);

and then just write:

executorService.submit(callable);

This would execute the callbale call method in async way, You can pass callback to the

callable object which would later be called to send the response back.

Captcha Hack - Part 1

Captcha is a type of challenge-response test used in computing to determine whether the user is human.





For example:









Does Captcha word can be read by a non-human eyes?

The answer is yes, In this Post I would explain the idea of how this can be done, the algorithm behind it and at last a sample code which implement the algorithm.

Trying to read the letters using image processing is pretty difficult, we would use different technique.

Not all Captcha words can be hack, However the hacking of Captcha code take advantage of the disadvanatges of poor Captcha words.

Many Captcha words are combine of four letters or digits,
Lets assume that it combine both from letters and digits, since abc containts 26 letters and there are 10 digits there are 36 combinations for each letter/digit.

This gives us a total of 36*36*36*36 = 36^4 = 1,679,616 combinations.

If the Captch word is combine only from letter it is even better, this would give us a total of only 26*26*26*26 = 26^4 = 456,976 combinations.

We can then build a DB with all the possible comibnations and give a unique key for each of one of the combinations.

When a Captcha image is displayed we then generate a unique key from this image, and find the matching one in the DB, this would give us an understandable word which can be used to enter the Captcha word.

Some assumptions were done to implement this algorithm:

• The Captcha word generate must be at the same format, meaning captcha background must have the same pattern, Captch words must be the same font each time.
• The number of letters/digit can't be over 4 digits, this algorithm wouldn't be able to solve the Captcha word since 36^5=60,466,176 which is too much to store at a simple DB.

The algorithm would be able to solve this special scenarios:

• If the number of digits is changing between 3 or 4 digits
• If the backgorund color or the font colors are changing
• If there are simple lines on the Captch word which intend to interrupt reading the word

In the next posts I would give a more detail algorithm with a solution for the above speical scenarios.

Webmasters - Improve your Captcha words!!!

Debug Web Applications

Debuggubg Web Applications is not an easy task,
However this area is involving, and there several ways now to debug Web Applications.

1. The simplest way is to use remove tags inside your html/jsp/jsf files and check the result, or adding border="1" to tags and see the layout of your html.
2. in FireFox you can use tool which is called FireBug, This is a great plugin in FireFix which allows you to debug JavaScript code, It can also show you the border of each component in the HTML, you can also see the HTTP Get and Post requrest between the Client and the Server.
3. If you develop your Web Application using Google GWT you can debug your Web Application in Develop Mode as it is a pure java application.

Typical Software Development Project Cartoon

80% of commercial apps to use open source by 2012

I had found this intresting article which claim that 80% of commercial apps to use open source by 2012, This are not good news to application software giants such as Oracle, Microsoft, Sun, IBM etc', While for every aspect of software developing you have Open Source alternative the Open Source alternative becomes more appealing for it's robutness, and for it being free.

In IDE enviroment when coming to choose between Eclipse, IntelliJ, Netbeans most choose Eclipse, In Application Server when coming to choose between WebLogic, WebSphere, JBoss and some companies use JBoss.
Tomcat, Apache, GWT all of then are open source.

So how open source projects make their profit? The answer is from support, and not only Open Source projects, many of the software applications giants make their profit from support which can reach up to 80% net incomes.

Only in companies in which the security is critical such as Banks or Goverment institutes Open Source is not being use beacuse it might containg backdoors.

Design Patterns Quick Reference in two pages

I had found this amazing Design Patterns quick reference which containg All the basic design patterns in only two pages!!!

I had printed it in my office and it is very helpful when designing new components,
This great quick reference can be found at mcdonaldland blog.

Enjoy

2008 Acquisitions

After Microsoft bid on Yahoo for 44.6-billion-U.S.-dollar and after Oracle bought BEA and Sun bought MySql 2008 seems to be the acquistion year, the question is what would be the next big acquisitions in 2008?

Here is my bet on the next big 2008 acquisitions:

• Salesforce.com would be bought by either Oracle or SAP. Both of this software companies want to extend their On-Deman CRM services, Salesforce would put them on the front line, After Oracle bought People Soft and and Siebel it would want to extend it's CRM products even more.
• RedHat would be bought by Oracle. RedHat which posses JBoss would likely be bought by Oracle which would very interest in the Linux and to extend it's middleware products.
• Microsoft would buy Logitech to extend it's computer control devices
• IBM would continue with it's straretgy to buy only small Start-up companies

Google Interview Questions - Part 9

Q1: How you would tweak bits in C to find out if a machine’s stack grows up or down in memory?

Q2: There are 8 stones which are similar except one which is heavier than the others. To find it, you are given a pan balance. What is the minimal number of weighing needed to find out the heaviest stone ?

Q3: N threads .. n resources .. what protocol do you use to ensure no deadlocks occur?


Here are the answers:


A1: Instantiate a local variable. Call another function with a local. Look at the address of that function and then compare. If the function's local is higher, the stack grows away from address location 0; if the function's local is lower, the stack grows towards address location 0.

A2: Divide the stones into sets like (3,3,2). Use pan to weigh (3,3). If the pan is remains balanced, the heavier is in the remaining (2). use pan again to find which is heavy from (2). (Total pan use 2) If the pan does not remain balanced when weighing (3,3), pick the set of stones that outweigh other. Divide it into (1,1,1). use pan to weigh first two. It it remains balanced, the remaining stone is the heavier, otherwise the one outweighing other is heavier(total pan use 2)

A3:

• Avoiding Deadlocks with Lock Leveling, each thread would have different level, thread with high level would be able to lock thread with lower level but not contrary.
• Avoid acquiring more then one lock at a time.
• Acquire the lock always in the same order.
• Shrink synchronized blocks.

Most Popular Developers Interview Question

This is one of the most popular developers interview question I know,

I had been asked about this question so many times.



The question is about the singleton design pattern, how to ensure a single instance of singleton object.



This answer is wrong:



public MySingleton getInstance() {

if (instance==null) {

instance = new MySingleton();

}

return instance;

}



This is because if two thread would call getInstance the same time then they may both enter the if statement before instance was initialized.



Another answer for this question is to initialize instance in the creation of the class.



private static final MySingleton instance = new MySingleton();



but if the classloader is not lazy then this class would be initialized even if no one use it.



A good solution is to initalize it in a static section of the class



static {

instance = new MySingleton();

}



this code would only be called when someone create an instance of the class and two threads can't enter the same time the static section.



Another bad solution is to write



public synchronize MySingleton getInstance() {
if (instance==null) {
instance = new MySingleton();
}
return instance;
}



This is not good solution since each time someone would get instance it would synchronize the call this would decrease the performance.



So a more sophisticated good solution is:



public synchronize MySingleton getInstance() {

if (instance==null) {

synchronize {

if (instnace == null) {

instance = new MySingleton();

}

}

return instance;

}



in this case two threads can enter the first if statement in the first call but then only one of them would get into the second if since the synchronize keyword, this would create only one instance, and also this would create in lazy when the singleton is first access.



Be prepared to be asked about this question in your next interview.

Spring vs. EJB 3

This article would overview the differences between Spring Framework and EJB 3

EJB 3.0

• EJB 3.0 is a specification.
• EJB is an architecture
• The EJB 3.0 framework is a standard framework defined by the Java Community Process (JCP) and supported by all major J2EE vendors. Open source and commercial implementations of pre-release EJB 3.0 specifications are already available .

Spring Framework

• Spring is an implementation.
• Spring is application framework
• The Spring framework is a popular but non-standard open source framework. It is primarily developed by and controlled by Interface21 Inc. Spring can work standalone or with existing application servers.

EJB 3.0 Advantages

• EJB 3.0 specification horroble old entity beans have been replaced with the JPA
• Deal with transactions and both succeed modularizing this functionality in a way that it doesn't clutter business logic
• Tune the transaction propagation levels via annotations
• Support state management via stateful session beans,
• Support injection of primitive types, allows injecting EJB's
• Supports AOP using interceptors
• Brings a basic implementation of component management

Spring Framework Advantages

• Integrates with many popular persistence frameworks including JDBC, Hibernate, JDO, iBatis and JPA.
• Transaction demarcation, propagation, and isolation are all declared through its AOP.
• Support state management via prototype beans and also request, session, and global session scoping
• Support injection of any type, allows injection of any domain objects using the features of AspectJ
• Suppotrs AOP by tightly integrating with AspectJ with all of it's features
• Spring 2.0 offers a lot more power and customization, but the use of AspectJ actually makes the hard to learn

Spring Framework Disadvantages

• The idea of annotations was to stop work with XML - Deploymenet Descriptor files which were in EJB 2.1, but Spring encourage a lot the use of XML for dependency injection
• Refresh the Spring load data is impossible, if you change something in the configuration you need to restart the Application Server.

You can integrate Spring and EJB 3 technologies to take advantage of their relative strengths and weaknesses, In Spring and EJB 3 there are a lot of duplication in the compoments you can choose the best of this technologies to use in you application.

Writing Cluster Application

Here is a checklist of things that need to be done to write good application which would work in Cluster enviroemnt

• When accessing the cluster from outside using HTTP or RMI verify that access to the cluster is done throught load balancer, this can be either software or hardware load balancer.
• Consider using 'Optimistic Locking' or 'Pemisitic Locking' if two or more clients may update the same row in the databse at the same time
• If you want to cache data all over the cluster and not on a single server then don't use Singleton, you would need to user Cache framework such as GigaSpaces or OSCacheor JCache etc', this solution is good mainly for read only data or data which is mostly read.
• In many application servers you can configure your MDB if it would run on a single server or on several servers.
• Avoid reading files/writing to files in the code, Since you may run on different hosts in the cluster and the files may not exists on the current host which you are running
• Use Synchronize where two threads may access the same data at the same time, This problem can also occure on non cluster system.

Please comment if you have other issues which are missing.