I recently set up a CI server for a new project at work, as the senior staff were quite keen on implimenting the tech I had recently pioneered on another project. After configuring the build, including automated nUnit tests, database, web and service deployments I was asked if I could integrate automated testing using SoapUI – something the test team had been pioneering for this project too.
SoapUI being a very lightweight, very powerful tool for testing Soap requests against a web service, such as an API or end-point. You can build a fairly expansive library of requests, configure the expected results and set pass/fail criteria based on those returning messages. So as a tool to test our new API, it was an obvious choice.
The SoapUI project had been created in its own SVN repository, so in TeamCity it was quite simple to create a SoapUI build configuration on its own, rather than stick it on the end of another build (although you can easily set the SoapUI tests to be triggered on success)
I won’t get into the whole TeamCity configuration because it is vast I’ll just stick to the SoapUI stuff. Before we begin, housekeeping so you know what I’m talking about:
- OS: Windows Server 2012 R2
- User Permissions: Administrator (Read, Write & Execute)
- TeamCity: 9.1.4 (build 37293)
- SoapUI: 5.2.1 (OpenSource)
I decided to install the full SoapUI package on the CI server, if you’re strapped for space I would imagine there is a way to use the testrunner as a standalone app – but the full package is only 250mb, so it wasn’t a huge impact.
In the installation directory “<install path>\bin\” there is a file named TestRunner.bat and TestRunner.sh if you’re a unix user. This batch file can be fed a multitude of arguments to be able to run your SoapUI project and generate results files. This is important since TeamCity cannot explicitly get the results while the batch is running (other than failure/success exit codes).
The TestRunner can be run directly through a “Command Line” build step in TeamCity, however I felt I needed more steps in order to organise and clear out space on the drive so I wrote my own Batch file which would do a little housekeeping, then run the TestRunner. Some people used MSBuild for this task – though I felt there wasn’t quite enough info for me to approach it that way, so I’ve gone old school and wrote a windows batch as follows.
@echo off
E:
IF EXIST E:\SmartBear\SoapReports\Results (
::Delete the output directory
echo "Reporting ouptut directory exists, deleting it"
RMDIR E:\SmartBear\SoapReports\Results /S /Q
)
mkdir E:\SmartBear\SoapReports\Results
E:\SmartBear\SoapUI-5.2.1\bin\testrunner.bat -rajI "E:\TeamCity\buildAgent\work\SWIM_TEST\SWIM-ware-soapui-project.xml" -f "E:\SmartBear\SoapReports\Results"
The idea being that the batch creates an output directory for the results, clears out any old files, then runs our TestRunner. The TestRunner as I said before takes a multitude of arguments for a variety of things. In this case I’ve used 5
- -r: Turns on printing of a small summary report
- -a: Turns on exporting of all test results, not only errors
- -j: Turns on exporting of JUnit-compatible reports, see below
- -I: Do not stop if error occurs, ignore them: Execution does not stop if error occurs, but no detailed information about errors are stored to the log.
- -f: Specifies the root folder to which test results should be exported
The -I argument was an important as when the TestRunner does it’s thing, if any of the tests fail TeamCity will pick up on the errors and mark the entire job as failed – this is obviously not the point in testing – we want detailed errors on the failures, not just a failed job – the I flag will remove some printing to the console but this can be pulled out later on from the test results.
After the -rajI command is the path to the SoapUI project, and after the -f argument is the path to where I wanted my results to go.
When I first ran the Job I encountered an issue where the build would not proceed beyond the “All Plugins Loaded” step – I searched the internet not finding much beyond others experiencing the same. I discovered eventually that the TestRunner was not capable of generating a soapui-settings.xml file and as such was asking me every time I ran the TestRunner if I wanted to send my usage data to SmartBear, while generally unrelated it was waiting for me to select an option, and blocking the rest of the job. The solution of which was to generate a settings file, which I did by launching the UI once. If you aren’t running the UI, I uploaded my sample config in a forum post here.
I also had a few issues in my error logs relating to the max_connections_per_host value as it was set to an empty string and Soap was trying to parse it as an integer. I had to manually update this setting to a numeric value. This can be done either through the front end or by manually editing your settings xml (Key HttpSettings@max_connections_per_host)
Edit through the front end using File > Preferences and under the HTTP Settings you can find the options.
So I now had my TeamCity Build config downloading the SoapUI from the server, and running the test cases – and I could see the output. Unfortunately all it was accomplishing was telling me it had successfully run – no actual test results. I was following a tutorial but it got a little vague about now on how to proceed.
When the TestRunner completes it generates a number of .txt files – these summarise the message and response from the server, as well as some header information and the rest of it. TestRunner also generates a summary .xml file, however this is just a top level summary of the execution. SO in order to get some proper reporting done you need to do 2 things. First, read in your XML results file(s), Second is to publish the failed tests as artefacts to provide greater detail on the failures.
For the first part – in your build configuration go to “Build Features” add a new XML report processing of report type “Ant JUnit” – since this is the type that SoapUI exports (due to the -j argument). You need to then add a monitoring rule to point TeamCity to the path where the report summary is kept. For me this was the following
+:E:\SmartBear\SoapReports\Results\*.xml
Adding this Build Feature will allow the after-testing report to be read into TeamCity, showing you the number of failed and Passed Tests. The only issue is that when viewing the stacktrace the information isn’t great – at best you will see the failed assertion from the test case. To improve on this we can read in the individual test outcomes from the reports folder. This is mentioned here, but it is rather vaguely worded so it took me a while to figure out.
Under General Settings in your build configuration you can add files to be captured as artefacts – these will be saved along with the build run so the physical files can be deleted, and the legacy artefacts can be removed depending on your TeamCity Configuration: To capture the artefacts add the path with a wildcard for *FAILED.txt – this is because every failed test in SoapUI produces an outcome file which ends with that string, so you can capture the details of the failed tests relatively easily.
Now when you run your SoapUI build, the tests will run, the top down failures and passes will be displayed and you can check your artefacts for detailed errors, including what message was sent and what message was received back. Very handy.
Sources:
- SoapUI – Running Functional Tests
- Running SoapUI Tests in Teamcity
- Integration with TeamCity – TeamCity waiting on a condition.
- Setting your preferences in SoapUI
Recent Comments