At the end of October of 2015 I decided to present a topic at the Boston Sitecore Usergroup, and after a few drinks, I brashly decided that I would present on Functional UI testing and Sitecore. One of my favorite tools to work with is Selenium, as I have worked with Selenium for about two years now and love tinkering with it. I also am a huge fan of Behavior Driven Design and tools such as Cucumber or Specflow. However, Sitecore or any modern CMS presents a number of obstacles to creating low maintenance automated browser tests. In order for automated tests to provide value beyond asserting whether or not a page isn't throwing an HTTP Error or exception, the test needs to rely on the content on the page. However, this creates a maintenance dependency from the seventh circle of hell. To make it even worse, common ways of modeling your tests against pages also crumble in the context of a modular CMS. As we enter into the Atomic age of Sitecore, we also need to up our testing game. So, with one too many drinks I made a promise only an intoxicated person would make, which was to create a better way to do functional UI testing with Sitecore.
In A Pickle
Gherkin is a DSL used for expressing business requirements and automated test steps. It uses a "Given, When, Then" structure to establish a context, action, and assertion. Runners such as Cucumber or Specflow then parse Gherkin to dynamically run steps. This allows our developers and business users to communicate more effectively and also ensure that our requirements stay in sync with what our code actually does.
Feature: A developer with a bad idea
Scenario: A drunk developer with a bad idea who likes to procrastinate
Given one too many beers and an idiot
When the idiot commits to a Sitecore Usergroup
Then panic shall ensue
Automated Testing and Sitecore
Automated browser testing with Sitecore presents a number of challenges, with most of them relating to the inability to easily access content outside of the Sitecore context. This inevitably leads to writing brittle tests with hard coded content or using an API to access Sitecore content. While these techniques work in certain circumstances, they can quickly become a maintenance nightmare at scale. With these constraints in mind, I started to explore ways to access the context that would allow easy creation, maintenance, and management of tests. The approach should also be reusable between solutions, introducing as little custom development as possible. I ultimately decided to create a proof of concept using the rules engine.
Gherkincore is a proof of concept to bring Gherkin style testing into Sitecore using the Rules Engine. Let's take a look at the above scenario but this time in the rules engine.
Challenge one complete, at the very least it is possible to emulate the structure of Gherkin in the rules engine. The next was to find out if the rules engine could execute conditions in actions in the same fashion that Gherkin runners execute the code backing the steps.
The rules engine lends itself nicely to parameterizing our tests, and in fact it even gives us "syntax highlighting" and "auto complete" in a way.
We can use the Tree syntax in an action or condition to create parameter sets to assist in making our steps. This allows us to create more abstract steps and create reusable mocks for our tests. For instance, in our above example we have beers, but as it turns out I don't actually drink beer - I'm more of a Manhattan type of guy. Let's update our rule condition to be more flexible with a Tree.
Using this technique allows us to create test rich test mocks that are easy to maintain.
A Practical Use: Selenium
While our above example is certainly relevant, I'm not sure if we need to have rigorous testing to ensure that our scenario plays out as expected. With Gherkincore I have wrapped parts of the Selenium API in the rules engine to allow automated browser tests to be managed as content. Below is an example of this.
What this is actually doing:
- Starting a PhantomJS driver
- Navigating to the Home item in our content tree
- When the DOM is ready it is using the selenium API to get the first (and hopefully only) H1 element on the page.
- Then we are checking to see if the value of the text in the Title field on the Home item matches the value of the text contained in the H1 element on the front end.
This approach solves some of the major challenges of functional UI testing with Sitecore. The two major points of interest here are how we are able to reference the Home item and and how we are making our assertion. With this approach we no longer have to hard code a base URL to get to our home page, which means that when we switch between environments, our tests don't need to change. A better example to consider would be if a content author moved a page in the tree that was used for functional testing. If the test had the URL hard coded in it, it would fail, but with our approach we have eliminated this issue. The second interesting thing in this example is our assertion, as many times it is necessary for our tests to have some knowledge of our content, but in the event that a content author changes content that our tests rely on, they break. By having access to the item under test, we no longer have this issue, since our test is simply comparing the value in the field.
Knowing It Worked
Another key piece of GherkinCore is the reporting pipeline, as the results at the end of a test are pushed into a pipeline which can be extended to provide formatting, images, or other types of post processing. Currently a very rudimentary report generator is in place which simply dumps screenshots and descriptions from each step into an image bucket with items that have an RTE on them.
- Feature Description
- Scenario Steps (Rendered rule)
- Pass / Fail of Scenario
- Step description
- Screenshot from our WebDriver during the current step
Other Potential Use Cases
Automated Functional Testing for Personalization
One of the biggest challenges of creating personalized web experiences is the amount of work required by a content team to create and maintain personalization rules. However, as personalization become more and more complex we need to have an efficient way to verify that our rules and engagement plans are behaving as we expect. While Explore mode is a great tool for simulating a users experience, I am skeptical about it's ability to scale.
Testing Content provided by FXM Etc.
It seems like Sitecore is positioning itself to provide external sites via integrations and JS injection. Using the GherkinCore approach we can reference content stored in Sitecore but navigate to external web properties. This could also potentially be extended to things like ECM.
As I mentioned before, this is a very rudimentary proof of concept, however I am pleased with the effectiveness of this approach so far and will be releasing a simple but stable version of it to the marketplace in the next week. I would like to point out some of the major flaws / challenges that I see and would enjoy any feedback from others on pitfalls.
The driver runs on your CM server
This is one of the top to-dos to fix once it becomes stable. I'm planning on offloading this using Selenium Grid.
Currently gathering reporting details from each rule action is very clunky, as a TestStep object is newed up in each rule condition/action. One of the road blocks I hit was retrieving the text of an action/condition in its evaluated state while it was being executed. Since this doesn't seem to be possible, at least from what I could tell, I had to create a work around, essentially duplicating information already available in our condition/action items.