«

Oct 25

How to set up a Jenkins CI Server on Windows for Google App Engine and Selenium

It’s been a while… Today I’d like to show you how to set up Jenkins on your Windows server. This post is quite specific to Google App Engine (GAE) and Python as there isn’t much info available on how to get continuous integration going if you’re developing for GAE. That said, much of the setup will be the same for other types of projects so this should be of value to anyone looking to get Jenkins going on Windows.

 

Prerequisites

There’s a few different ways that you can run Jenkins. The simplest is to just download and run it, as this shamelessly stolen snippet from the Jenkins wiki explains:

“To run Jenkins, minimally you need to have JRE 1.5 or later. After you download jenkins.war, you can launch it by executing java -jar jenkins.war. This is basically the same set up as the test drive, except that the output will go to console, not to a window.”

Now assuming you don’t enjoy the pain and torture of doing something manually, you probably don’t want to have to worry about starting up Jenkins.. ever. This leaves you with two options: (a) use the java service wrapper, which is not recommended, as alluded to here; (b) deploy Jenkins inside a service container, which is the recommended way.
There are many service containers which you can use, but I’ll be using Apache Tomcat as it’s probably the most popular.

Important!

In short, you need:

To get Jenkins going:

To automate GAE unit and Selenium tests:

 

Tomcat Installation

Your best bet is to get the Tomcat 32/64-bit Windows Service Installer which makes the installation of Tomcat a breeze. When you get to the Configuration step of the installer you may want to change the default ports to suit your environment. Choose an admin user name and password at this stage to save a little time later on.

Once Tomcat is installed and started, simply point your browser at http://localhost:8080 (or whatever you chose as your Connector Port). You should see something like this:

Click on the Manager App button, input the admin user name and password you chose earlier when prompted by your browser, and you should see the “Tomcat Web Application Manager” page. This page lists the applications deployed to Tomcat. If you chose not to enter admin credentials during installation, you won’t be able to log in because Tomcat does not enable any logins by default. Simply cancel the prompt for credentials and you will be taken to a page explaining how to fix that (basically you need to find and edit the tomcat-users.xml file in your Tomcat installation path).

Before carrying on you may want to check that the Tomcat service is set to start automatically with Windows.

 

Jenkins Deployment

Download the Jenkins Java Web Archive (.war) if you haven’t already. Back on the Tomcat Web Application Manager page, head to the Deploy section. Under WAR file to deploy simply browse to choose the Jenkins WAR and hit deploy. Wait a few seconds and Jenkins should be deployed and running. Click on its path in the list of applications in the Tomcat Manager and you should see a rather smug butler (a.k.a. the Jenkins home page).

Home Directory

The default home directory for Jenkins depends on your environment. On my Windows 7 box it ended up at C:\Users\<user>\AppData\Local\.jenkins, but on Windows Server 2008 it ended up at C:\.jenkins. The reason I’m mentioning it is because we had an issue with long paths which was preventing our builds from succeeding on the Windows 7 machine. Moving the home directory to the root of the C drive fixed that issue. You can see where the home directory is, along with instructions on how to change it, by going to Manage Jenkins and then Configure System. I could only change the home directory by modifying the web.xml of the expanded Jenkins image (located at C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\jenkins\WEB-INF). I tried the recommended way – creating an environment entry in Tomcat called JENKINS_HOME or HUDSON_HOME – but couldn’t get it to work. Remember to stop Jenkins from the Tomcat web interface before attempting to change the home directory!

Manage Jenkins and Install Plugins

Before setting up your build job you may want/need to install a few plugins. CVS and Subversion plugins are installed by default, but Mercurial is not. We use Mercurial so that’s what I’ll be adding, as well as the Instant Messaging and Jabber plugins for some extra geek-joy (automated build notifications). After installing plugins you need to restart Jenkins from the Tomcat web interface.

When you go to the Manage Jenkins page you may see the following warning: Your container doesn’t use UTF-8 to decode URLs. If you use non-ASCII characters as a job name etc, this will cause problems. See Containers and Tomcat i18n for more details.
To fix this, simply edit the server.xml file at C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf, and add URIEncoding=”UTF-8″ to the appropriate <Connector> element.

 

Creating the Build Job

Now to set up the build job. Click New Job and give it a name; select Build a free-style software project and hit OK. You will be taken to the configuration screen for the new project:

Source Code Management

Select and configure access to your source code repository. For Mercurial all I had to do was input the repository URL. If your repository is authenticated you need to add the credentials to the URL. So if you’re hosting with BitBucket, your URL would look something like: http://username:password@bitbucket.org/company/project

Build Triggers

Set up what should trigger a build. In most cases you’ll want to choose Poll SCM. To poll every minute simply enter five stars with spaces (* * * * *) into the Schedule textbox. Otherwise check the in-context help for details on how to set up a more complex schedule.

Build

This is where you add your individual build steps. Remember to use Execute Windows batch command instead of Execute shell. NB: I’ve detailed the steps for GAE and Selenium further down.

Post-build Actions

Add email or IM build notifications. If you installed the Instant Messaging and Jabber plugins earlier you should already see an option for Jabber Notification. Note that for these to work, you need to configure them first by going to Manage Jenkins and then Configure System.

 

Making it all work with GAE and Selenium

So much for the basics of setting up a Jenkins server. Now we get to the good stuff.

You might have wondered why I’m using two different versions of Python. There are two reasons for this: Firstly, the Selenium WebDriver API does not support anything older than Python 2.7. Secondly, the Python 2.7 runtime for GAE is still experimental at time of writing. So if you’re feeling adventurous go ahead and try running everything under 2.7. If you’d like to play it safe instead, you’ll need to create a seperate project for your Selenium tests that compiles under 2.7. Later, when 2.7 is officially supported by GAE, you can merge this project with your GAE project.

I’ve assumed that you’re using GAE Testbed and the Selenium WebDriver API. If you’re still using Selenium RC I suggest you read the migration docs.

Installation

If you haven’t already, install the GAE Python SDK, Python 2.5 & 2.7 and setuptools on your build server (NB: you need to download a specific version of setuptools for each Python version, or use the ez_setup.py method).

Use easy_install to install the packages required for GAE Testbed in your Python 2.5 directory, as detailed here. You also need to install nose and selenium for Python 2.7
In summary:

C:\Python25\Scripts\easy_install-2.5.exe gaetestbed
C:\Python25\Scripts\easy_install-2.5.exe nose
C:\Python25\Scripts\easy_install-2.5.exe nosegae
C:\Python25\Scripts\easy_install-2.5.exe webtest
C:\Python25\Scripts\easy_install-2.5.exe BeautifulSoup

C:\Python27\Scripts\easy_install-2.7.exe nose
C:\Python27\Scripts\easy_install-2.7.exe selenium

Build Steps

Add a build step to your Jenkins project to run your GAE unit tests. It should look something like this:

C:\Python25\Scripts\nosetests-2.5.exe "C:\.jenkins\jobs\<Jenkins project name>\workspace\
<GAE project name>\<directory containing tests>" --with-gae --without-sandbox
 --gae-lib-root="C:\Program Files (x86)\Google\google_appengine"
 --gae-application="C:\.jenkins\jobs\<Jenkins project name>\workspace\<GAE project name>"

Build Step 1

 

For your own sanity I would suggest getting one thing to work at a time. Save the Jenkins project configuration and click Build Now. You should see a new build starting in the Build History pane. Click on the running build and then on Console Output to see the results and to troubleshoot any failures.

 

Next we need to get the Selenium tests running. This requires a little extra effort as we need to start and stop a local App Engine instance. We’ll automate this away in a moment, but for now let’s see that it works. Start GAE from the command line like this:

C:\Python25\python.exe "C:\Program Files (x86)\Google\google_appengine\dev_appserver.py" --skip_sdk_update_check
 "C:\.jenkins\jobs\<Jenkins project name>\workspace\<GAE project name>"

If you see something like┬á “INFO┬á┬á┬á┬á 2011-10-23 18:43:52,380 dev_appserver_multiprocess.py:637] Running application <name> on port 8080: http://localhost:8080” then GAE started successfully.

 

To automate this, create a batch file with the command above and use the schtasks utility to create a once-off task that you can start and stop as you see fit. Here is more info on the topic.

- Once off:
schtasks /Create /SC ONCE /ST 16:35 /TN RunGAE /TR C:\.jenkins\RunGAE.cmd

- Start and stop:
schtasks /Run /TN RunGAE
schtasks /End /TN RunGAE

The time parameter to the schtasks /Create command is insignificant, but it needs to be greater than the current time. Note that this will cause the task to run once at the specified time.

 

We can now add three more build steps to our Jenkins project – the first will start GAE, the second will run the Selenium tests, and the third will stop GAE again. Here they are:

schtasks /Run /TN RunGAE

Build Step 2

 

C:\Python27_32\Scripts\nosetests-2.7.exe "C:\.jenkins\jobs\<Jenkins project name>\workspace\
<GAE project name>\<directory containing tests>"

Build Step 3


schtasks /End /TN RunGAE

Build Step 4

 

Update: This last build step won’t run if the build fails, but we still want to stop GAE. To fix this, install the Post build task plugin (remember to restart Jenkins afterwards) and configure your build project accordingly. You’ll need to delete build step 4 as shown above and instead check Post build task under Post-build Actions. Match on log text of ‘SUCCESS’ or ‘FAILED’ or ‘FAILURE’ or ‘ERROR’ so that this task always executes and paste the command from build step 4 into the script textbox.

 

That’s it! Please do comment if you encounter any additional issues or even just to say that it works.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>