INTRODUCTION

Oleeo strives to ensure that we deliver the best quality possible to all of our customers at all times in line with our quality policy. As part of our efforts in ensuring we deliver consistent high quality, we utilise configuration testing as part of our service delivery process. Our Oleeo implementation consultants test the configuration of solutions against agreed scope, requirements and where appropriate, specification documentation contained within the corresponding sugar case. When the configuration under test has achieved the approval of the implementation consultant they will present the tested solution to the requestor thereby initiating the User Acceptance Testing (UAT) stage of the service delivery process, during which the requestor or testing resources on the customer team will perform UAT testing against the configured solution.

Oleeo incorporates a UAT stage as part of the service delivery process to enable our customers to ensure that the configured solution meets their business requirements. The UAT stage is an opportunity for our customers to test the configured solution for themselves, see it in action before it is deployed to the live environment and decide if the solution is going to effectively do its intended job for the customer's company. It is extremely important to Oleeo that the solution provided fits all requirements defined within the scope of work and that it is the overall best solution that Oleeo can provide.

What is UAT?

UAT or User Acceptance Testing is the part of the Oleeo service delivery process during which customers test and review solutions which have been approved by the Oleeo implementation consultants to ensure that they meet the requirements set out in the corresponding case.

 At this stage of the process, customers will test the solution that Oleeo has provided and either sign the work off, ready to be deployed to live, or raise any feedback, issues or general queries. 

What is the value of UAT?

UAT allows for customers to add additional value and get hands-on with configured solutions, ensuring these effectively meet their day to day business objectives. Testing this way creates an opportunity for customers to identify issues or deliver feedback that implementation consultants will not have noticed due to them not knowing exactly how a certain user operates within the ATS in their day to day role.

Customers can achieve this by making sure that the configured solution works per the agreed scope but more importantly they should be using the UAT stage to review whether the solution is the most effective way to meet their business’ needs. 

The UAT phase also offers the ability to leave any feedback regarding the configuration you’ve tested or to raise any additional requirements that may come to your attention. If these new requirements are within the scope of work they will be accommodated or alternatively, may incur an additional cost if they are outside of the agreed scope of work.     

SELF-CONFIGURATION

What is self-configuration?

Any change that is configured within the ATS by Oleeo’s customers is self-configuration. The process for self-configuration does not include any Oleeo resources until the point that the change is pushed to the live environment. This also means that the implementation consultants do not test any configuration produced during this process, so it is paramount that customers are testing their own self-configuration prior to raising a self-config push request.

If you do not have self-configuration privileges but would like to explore the option, please reach out to your Customer Success Manager.

Don’t forget to add your locks!

Self-configuration changes, similarly to Oleeo configuration changes, require a lock on the ATS system in the configuration environment prior to commencing any changes. It is crucial that customers add a lock for any planned configuration changes before making the changes, not during or after. 

Not adding a lock risks incomplete configuration being deployed to the candidate-facing live environment as there is no way of identifying what work, if any, is being done in the background. The locks on a system are a safety precaution that allow everyone to know that there is work in progress. 

Configuration locks are critically important and contribute to awareness of all existing work when evaluating whether a system can be safely pushed to the live environment. If there are no locks, the Oleeo team has no way of knowing if there is ongoing or incomplete configuration in the system. This could have a negative downstream impact on the customer’s system due to the risk of incomplete configuration being pushed to the live environment.

Requesting a push to live 

To request a push to live of self configuration changes, raise a case in Sugar with the case type set as a self-configuration push request. Within the details of the case, you should copy the description of your lock straight from the ATS into the case details. This will enable the Oleeo team to know exactly what lock you are referring to and avoids any confusion that may delay your push to live.

Do note as well that you do not have to raise a new case for every single lock. If you have multiple pieces of work that you are requesting pushed to live at the same time, you can copy and paste all of their lock descriptions into the one Sugar case that you raise.

Once the Oleeo team has reviewed your self configuration push request they will add an update in the case advising either that system has been pushed to live or, if there is something blocking your push they will include details of what the blocker is.

If there is something blocking your push to live, please refer to the lock referenced in the note the Oleeo team have provided. The lock mentioned within their note may require your attention.

WHAT DO I NEED IN ORDER TO TEST?

Always consider your approach prior to testing the newly configured solution. Ask yourself; what is required for this test? A test will usually consist of a few different things:

Understand what you’re testing

You first need to know what it is that you’re testing, once you know what you’re testing you will be able to better consider where in the ATS your test is going to take place and what kind of test data you’re going to need. For example, if we’re testing an application form, we know that we will need to create a vacancy with that form attached. On top of that we will then need to apply to a freshly made vacancy in order to access our application form via the job board.

Test data

You need to create test data. Ensure your test data is created in the configuration environment. Also use naming convention to easily identify your test data in the ATS, an example of this is outlined in our do’s and dont’s section. Your test data should be specifically relevant to the item that is under test. You want to have identified all possible results your test may produce and create individual pieces of test data to accommodate all of these possible results. Your goal here is to achieve all of those expected results with the corresponding test data. Good tests rarely ever test just one outcome so you may need more than one instance of test data. 

A way to validate your result

Once you’ve conducted your test you must validate the results. Validation in this context is checking the accuracy of your test result. This is just as important as the test itself. Validation can be done by checking a test result against some form of specification document or checking testing notes that your implementation consultant has provided in the testing tab of your corresponding Sugar case. Validate against business requirements by testing against a use case situation that will reflect your day to day activity using that configuration. Testing outside of an expected result is also a form of validation which will help to ensure that the configured solution covers as many use case scenarios as possible. 

Business requirement

Aside from the configuration working as it should, it is paramount that the solution provided is also meeting your business requirements as effectively and efficiently as possible. Do keep this in mind as you test whilst considering how you might use the configured solution in a day to day situation and try to encompass as many of these different possibilities into your test as you can. Consider possible future proofing, downstream impact and the solutions versatility. If you feel the solution can be improved once you’ve concluded your testing or if you have an additional requirement that has come to light, do raise this within your UAT feedback on the Sugar case. Any new requirement that is within the original scope of work can be included in the existing Sugar case. If the new requirement is outside of the agreed scope of work then a new case will have to be raised which will likely incur an additional cost. 

TESTING DO’S AND DON’TS 

DO - Always create new test data

At the start of every test, you should be making fresh test data as a general rule of thumb. In the majority of cases you will need test data for any test that you're conducting within the ATS and it should be fresh data every time so that you know your test result is uncontaminated and not producing false results . 

DON'T - Reuse old test data for a new test 

This is because you have no way of knowing for sure if anything in a piece of test data has been altered. Altered data may produce a result outside of your expected outcome and result in a false positive or false negative result. For example if you’re testing forms with old test data there may be a field that has been selected within that data that may hinder your ability to see a certain aspect of the form you’re testing. As a result you may report that the test has failed but in reality it is because the re-used data didn't possess the specific criteria to test your form effectively.

DO - Conduct tests in the configuration environment 

This is the safest place to test and for any changes to be configured. Any and all solutions should be configured and tested in the configuration environment as it has been designed to reduce risk as it is away from potential candidates that operate within the live environment.

DON’T - Conduct tests in the live environment

Test data created in the live environment comes with a vast amount of potential risk. Test data in the live environment carries the risk of being visible on job boards to potential applicants for example. This can result in confusion for applicants and lead them to be discouraged from exploring your recruitment portal. Testing in the configuration environment carries little to no risk as opposed to testing in live and, for this reason, should always be used over the live environment.

Once your system has been pushed to live, you do not need to re-test in the live environment. The live environment is a replica of the configuration environment at the time it’s been pushed to live meaning the solution that you tested in the configuration environment will now exist as an exact copy in the live environment.

DO - Make sure you test the candidate experience 

Conduct your test the way a candidate would experience the ATS with any configuration that’s candidate-facing. Your primary focus is the candidate experience so it is important to see what they will see. The candidate facing side of the ATS is one of the most necessary places to be meticulous in your testing due to the risk that comes with that level of visibility to potential candidates.

A few examples of how to test the candidate experience are:

• Apply to vacancies via the job board rather than adding candidate manually to vacancies
• Utilising the log in as candidate feature to view the candidate portal
• Generate forms and letters on the candidate end of the ATS or generate them against an application and view them in the application history on the recruiter end
• Send any emails to yourself using an email alias

DON'T - Test using shortcuts 

The hard way is often the right way. Ensure that you are testing how the newly configured change will operate as closely to actuality as possible. Your goal is to ensure that you are replicating how the configuration under test is going to be used in a day to day situation.

For example, if you are testing a process, you should always move the candidate naturally through the process during your test and avoid using the “other status” feature wherever possible. This feature carries the risk of not running decision actions and autoscores that could impact your result. These status actions would run if you moved through the process naturally but using the ‘other status’ feature introduces the risk of them being missed!

Be sure that you’re adding any new forms to a relevant vacancy or application. From here log in as a candidate and view the forms this way, not via the form editor. You will see the forms as a candidate would, which is how you should be testing.

Also, anything in the ATS that can be generated; forms, emails, etc should be generated against an application or sent to the tester using the testing alias described in the paragraph above. This ensures that all candidate facing information is being received correctly as you are putting yourself in the candidates shoes during the test. Not generating these items against a candidate will not test that they’re sent to the correct recipients or whether any visibility rules and data paths are behaving as expected at the time a form / correspondence is generated.

SUGGESTED UAT PROCESS - THINGS TO CONSIDER

The goal of User Acceptance Testing is to assess whether a configured solution that you’re being provided can support your day to day business and user scenarios whilst also ensuring the solution also performs its intended function. Below is a suggested process of how to achieve this. This process encompasses the basics you will need to address to get the most out of your UAT.

1 - Planning approach

The UAT team should consider their top level approach to their UAT testing. Things that should be considered in this phase include:

• Testing time frames - You want to have a high level idea of how long your test will take so that you can prioritise workload and be in a position to communicate to the Oleeo team early in the event of a necessary push to live. Account for any re-tests and potential additional configuration as these may become a factor and extend the time needed for your testing process.
• A test plan - Map out the steps you will take to test the solution in real world situations. By understanding what your test will include, it will highlight where in the ATS you need to operate and what test data you will need to conduct your test.
• Test more than once - Plan to conduct your test a few times with altering variables wherever possible, this will be a truer test of real world situations due to you encompassing more potential outcomes into your test. This is a good test of what the configured solution will see in day to day usage.
• Have an expected result in mind - You will need to have a general idea of what your test should be achieving and what the expected result of each test should look like. Your test should prove that the configuration is functioning as intended and that the configuration is meeting your business requirements.
• Have a way to validate your test result - A specification document is the best way to validate a test result. Specification documents are defined by the job requirement provided in the scope of work. This is your safest and most effective form of test result validation. Reach out to the implementation consultant who configured your solution if you do not have one. For self configuration Oleeo recommends creating some form of documentation to test against. This document can also serve as a point of reference for the future if you want to look back on any configuration you’ve done.

2 - Test data

You will need to set up any data necessary for the completion of your test. Do not re-use existing test data for any of your tests. 

• Create test data that will encompass the situations you have laid out in your planning phase. Ensure you are using new test data so that you don’t risk invalidating your test result. Creating all of your test data prior to an extensive test will allow for a better view of the overall workload and will save time in the execution of the test execution phase.
• Remember your naming convention for your test data.

3 - Test execution and documentation

Now that you’ve composed your test plan and created your test data, you can conduct your test. Ensure you document any and all findings as you go. This may include screenshots or ID numbers from the ATS so that your data can be quickly located if it needs to be revisited.

Make note of all data, whether it passes or fails its test and anything of note that you discover during your test. However you go about documenting your test results and findings, ensure the information is concise and accurate. Anyone should be able to review your notes and be able to replicate the test you’ve completed.

• Conduct your test using your test plan and your test data. 
• Do not hesitate to conduct random exploratory tests if you become aware of new variables or issues that come to light during your test. If you come across anything that you feel may produce an outcome that you have not initially accounted for, create a piece of test data that includes that new variable and test what result you get.
• As you test, ensure you document all of your findings. Ensure that you document all instances of test data passing and failing during testing. Documenting what test data passes and fails allows for comparisons between any test data that may be producing different results. This will also assist in any troubleshooting or root cause analysis you may be doing when trying to identify why the results you’re seeing are different.
• If your test fails, make a note of the specific data you used for that test and specifically make note of that test data’s ID number. Test applicants and vacancies will all have an ID number and so will the majority of items that exist in the ATS. Referencing these ID’s in your notes will allow for your test data to be easily located within the ATS. This will give them a better chance of replicating your result and fixing any issues that may have arisen during your test.

4 - Feedback phase

This is the final phase of your UAT process. Here you will communicate any test failures and suggested improvements that you identified in the previous phase within the sugar case that relates to the work that you’re testing. This is also an opportunity to add any additional notes or requirements that may have come to your attention during your testing period.

• Failed test results and any improvements that you identified during testing should be communicated in the notes section of the corresponding sugar case. Be as clear and concise as possible with the information you provide to ensure that the person attempting to fix any issue you’ve raised can easily find your test data and / or replicate the steps you took to test.
• If you would like to raise a new requirement at this stage and it is within the scope of work, make a note of this and it can be configured alongside any fixes that may be required. If the new requirement you raise is outside of the existing scope of work for that particular case, then a new case will have to be raised for your new requirement which may incur an additional cost.

5 - Plan to pull a configuration report

Once you’ve completed your UAT and your changes have been pushed to live we recommend that you download a configuration report from your system so that you have up-to-date specifications. 

• This is most valuable after a push is complete as you want your specifications to match your live environment. 
• Your system Super User should have the ability to pull these config reports. 
• Pulling specifications from your system will give you a point of reference for all existing configuration in your system that existed at the point of your last push. 
• This report can be used as a specification going forward and can also be used to refer back to if you experience any issues whilst configuring any new changes.