My Journey into a QA Security Mindset: Engage your solution with multiple R&D teams
Thanks for joining me for my second post in this QA Security Mindset series! For this blog post, I will focus on the steps I took to create an automated solution for testing website vulnerabilities. If you haven’t read my first post, you can find it here!
Gather Information
When I started at AppsFlyer a year ago, my first mission was to understand the company and the structure of our QA department. At the time, we had 400 developers, and a group of 40 QA engineers.
As always, to gather useful information I needed to approach the key people from our QA group. My main goal was to create a solution that will be easy for them to implement and allow us to rapidly scale our security testing, with minor impact on their day-to-day testing stability and maintainability. I set out to answer the following questions and over time learned how things in our company work:
Q) What architecture does our Appsflyer product have?
A) In a nutshell, our product architecture includes hundreds of microservices, web handlers behind AWS LBs and Kafka as our message broker. Our dashboards are powered by click-house and druid and we use S3 for storage of processed data (By Sparks) and reporting data.
Q) How are our R&D organisation groups separated?
A) Our R&D organisation includes 3 main pillars/core groups along with several squads. Our 3 core groups are Analytics , Real Time Attribution and Marketing OS. The rest of the products are grouped as squads and stand alone units ( i.e : OneLink, Audiences, ROI360, P360, etc).
Q) How are our R&D QA groups separated?
A) Our QA groups are embedded into our development groups and not as a separate pillar.
Q) How do the quality engineers develop and operate their testing?
A) The majority of the QA teams develop their testing in Python and use pytest as a framework. As part of the framework, the entire group is using a common infrastructure code base, in-house web services which run on AWS, 3rd party frameworks and open source software. The operations and executions of the testing are invoked by Jenkins and run on AWS instances as part of CI or part of stand alone on-demand requests. In addition, as part of the development and design, testing is required and run as a monitor on the production system under the QA teams ownership.
Q) What Types of testing are used in QA tests?
A) As Appsflyer products include many sub systems, our quality engineers focus on several testing types and monitoring activities such as API testing, Selenium using Selenoid for front end testing, A/B testing, Functional, Data permission, integrity and accuracy, Mobile Testing, SDK testing over Mobile web, Mobile App and more.
Q) How are automated tests managed?
A) Each team has their own gitlab repository which includes tests for their own product
Q) Has anything similar been implemented previously?
A) A few QA groups had implemented some security tests, but they were very dedicated to their own products and were not cross-group.
Playground Time
After I had collected all this information, it was time to start playing. This stage is vital! In order to understand the world, you have to play in it!
I chose a service and I started to evaluate and explore with a couple of public http endpoints sending several requests to the server and investigating the responses.
I started to look for common parameters that are used with different services. For example, we have an email parameter in the request, whether in the request query string param or in the body.
I saved a list of all those common parameters that I found for later.
Let’s start creating our solution
Having collected information and built an understanding of how everything works together in QA, it was important to make a decision on how to build a solution that would have an impact with all of the QA teams.
There were a few options on the table, such as writing my own security tests for each product, but this was not very scalable. Also, each QA team knows their own services better than I do. By creating a solution that would have the QA teams write their own security tests, we could increase coverage more quickly and efficiently than I could if I did it myself. Moreover, the purpose is not only to increase coverage but to change the mindset and provide accountability into our teams.
Ultimately the decision I took was to build a Python library which other teams could import into their own projects.
When creating a solution as a library you might need to consider the programming language that you will use. Since most of the QA teams work with Python and I wanted it to be easy to integrate with them, I also used Python with the pytest framework.
Side remark: There are cons and pros of using a library. I will be looking into using a service in the future, which would have the benefit of being able to write in any language I would like. I will be writing about this in a future blog post to compare between the different methods for including security tests in an already functioning QA group. However, for now, a Python-package which can be imported into existing QA group projects, has made the most sense.
In order to visualize how my library would be used by different teams, i created the following diagram:
With this architecture in place, I had to decide which type of testing I would implement. The natural choice is to select API testing!
Reasons:
- I am focusing on HTTP requests to the server
- Easy to implement
- API tests are easier to maintain compared to UI tests
- Quick run time
I had to create something generic that will fit every team and that would be able to test different services in a microservices architecture. For that I created an object in Python with properties that represent all the required parameters that can be used to send HTTP requests:
Request method: The request method, One of GET/POST/DELETE/PUT (There are additional HTTP methods but currently I need only those)
- Request path: Without the Domain to keep it dynamic for use on different test environments and production.
- Request body (payload): The request body, in case the request method is one of POST/DELETE/PUT
3. Request authentication type*: We have different ways of authentication in our system. I had to support it all. (different kind of tokens , JWT. cookie ,etc)
*Website authentication is the security process that allows users to verify their identities in order to gain access to their personal accounts on a website.
4. Request headers: The request headers.
In the Playground Time section I mention common parameters between services (for example an email address). I had to find a way to allow the QA engineers to specify which request parameters they wanted to test. I instruct them to mark parameters with double curly brackets and then the security package will replace them with values using the Jinja2 package (templating language).
Let’s look at a request as an example:
GET https://www.mywebsite.com/foldera/file?email=example@email.com¤cy=USDA test case will look like:
def test_idor_myendpoint():object = TestedEndpoint(
request method=’GET’,
request_path=’/foldera/file’,
request_query_string_params='email={{email_param}}&
¤cy=USD',
request_authentication_type=’auth1')runIdorTests(object)
Then upon running the test, the security package will replace the email_param with values from the security package itself.
Proof of concept (POC)
Any solution that is used by several teams should take one small step at a time with short deployment cycles for improvements and quick feedback loops.
I coordinated a POC with 2 teams to evaluate our solution and objectives before spreading the news. We decided to start with an IDOR vulnerability. IDOR stands for Insecure Direct Object Reference and it means that a user is able to access data that they shouldn’t be able to access. Join my next blog post to learn more about IDOR vulnerability. In short, the easiest way to find IDOR bugs is by manipulating values in the request to the server. For example, let’s say we have the email parameter in the request, we can send and manipulate this parameter with different values:
- Empty
- Other account email
- Non existing email in our system
The assertion for the tests is needed to make sure that none of those invalid requests return response code HTTP 200 OK. Getting this response code for such requests might be a sign of an IDOR bug.
Additional value of this solution is that the quality engineer declares who is authorized to access the tested service and the security package will know to build dynamic use cases also for non-authorized entities.
The security package is able to do this by having predefined accounts and users, with their configuration and settings. The accounts configured in the package are not authorized to access each other’s resources.
Another objective I defined as part of the POC is to focus on public endpoints whether it is from UI or API. If it was from the UI, then we opened the relevant page(s) and performed all the actions from this page whilst using a sniffer in the background to see the requests to the server. Such sniffers can be your browser’s development tool or any web proxy tool such as Burp or Fiddler.
As the POC continued and more endpoints were added, the number of parameters that I had to support got bigger. One of the ways I evaluated my solution was that adding additional parameters didn’t require a lot of effort and was able to be done quickly.
We are going live!
After the POC phase and completing our objective, I published the security_testing package to our artifactory. Each QA team can import the package using pip to their existing testing repository and start writing automated tests that focus on IDOR vulnerabilities. It was a great feeling to see how quickly the teams can adopt IDOR vulnerabilities as part of their services testing (new or existing) and create visibility to R&D. This was achieved with hard work and a lot of effort from people within the QA and Security teams.
It was exciting seeing it working and finding bugs, some were already known and some were NEW!
Thanks for reading, I hope you join me for my upcoming posts where I plan on writing about:
PART III: My Journey into a QA Security Mindset: IDOR vulnerability
PART IV: My Journey into a QA Security Mindset: SSRF vulnerability
PART V: My Journey into a QA Security Mindset: Information disclosure vulnerability
.
.
.
PART X: To be decided
