This post is part of the Pride & Paradev series.
Should you use production data or generate test data for testing?
You should generate test data for testing
Generating test data is the only reliable way to accurately run tests repeatedly and consistently knowing that the input test data hasn’t changed.
Some applications rely upon specific data which is either hard to find, or hard to fake. For example, the web application I am working on displays different promotions based upon which day of the week you are using the system, and also changes prices depending on the day of week and time of day.
If you were using production data for testing, you would either have to run tests at specific dates/times to test different promotions/prices, or you would have to change the server date/time to test these. Changing the date/time on the server will effect anyone else using that server, so should be avoided. It also means that as you run your automated tests continuously against new check-ins, if you don’t use a known set of generated test data, you will get different results depending on time of day.
When developing an entirely new feature, there won’t be production data that you can use for testing, so you will need to generate some in this case.
Generating specific test data will often take longer sourcing production data, but will retrieve results over time as tests are run very consistently against a known data set.
You should use production data for testing
When you’re testing a web application, you’re as much testing the data as testing the application behavior. Using production data will ensure that what you are testing will be as close as possible to the actual behavior once the feature is released to production users.
If you generate test data and use it to test, who is to say that this test data is actually valid. If you generate test data through lower level means (such as SQL insert scripts), you may introduce test data that isn’t representative of that in production that may either introduce errors in functionality when actually running against production data, or errors in test that won’t actually exist in production. As your database schema updates and evolves, you will need to also keep your data generation scripts up to date so they are reflective of production at all times.
If you do use production data, you need to be clever about how to source data. Querying the database using SQL scripts is an effective approach as it will enable you to quickly find real data that you can use to verify a story has been implemented correctly.
It will also allow you to identify outliers and edge cases that can be tested using real production data against the system in development.
If there any concerns about using production data for testing, these can be mitigated by obfuscating the data so it is indistinguishable.