Presentation: Git Gud with Property-Based Testing
Share this on:
What You’ll Learn
-
Learn how property-based testing can improve the quality of your code base.
-
Hear how Polysync is using property-based testing in place of integration tests.
-
Understand that capabilities and limitation of property-based testing.
Abstract
Traditional unit tests force developers to make difficult design decisions on their own, meaning their tests can be only as robust as the human imagination allows. In searching for an alternative to this limiting style of testing, I’ve found Property-Based Testing as a meaningful alternative. Property-Based Testing presents a marked shift from this typical awkward style of test creation. Instead, test cases are programmatically generated to try to explore not only execution path coverage but also the domain input space. This bolsters the possibility of exposing unidentified problematic inputs and states.
Typically, Property-Based Testing is applied close to the code in order to exercise functions. I’ve found that it can also be meaningfully used for testing at the integration level. Over the course of my discussion I will cover my experience using both methods as a way to find casting and overflow bugs in PolySync’s Open Source Car Control (OSCC) project, as well as for work involving the validation of security research and hardening strategies for git repositories.
What is the focus of your work?
Today, I am working on using property-based testing to create various different kinds of distributed git repository scenarios. The goal is to validate that our solution to the git metadata vulnerability is working as designed. So I am trying to create any kind of wacky system state to ensure that our solution still works.
You work for a Self-Driving-Car System Company. You are talking about git and you are talking about property-based testing. What’s the connection?
We are trying to secure our deployment tools so we can ensure that nothing is being tampered with. This git test work is based on a security vulnerability that was exposed in a paper (On Omitting Commits and Committing Omissions: Preventing Git Metadata Tampering That (Re)introduces Software Vulnerabilities). We are trying to make sure our repositories are secure, and we are delivering what we are saying we are delivering in a way that does not mess up the data.
What do you hope people get out of the talk?
I’d like to let people know that property-based testing is a really great way to enhance your testing While property-based testing seems like a scary concept to get into, if you can learn how to correctly use it your tests will be more robust and more interesting to develop. It also provides more test cases for less work than traditional testing methods.
Is this a code talk or a case study talk, are you showing people how to write a test or how to implement a program?
It is pretty much just going to be based on my experiences learning property-based testing. When I started testing, I really was not familiar with testing much at all. I did some unit testing and cucumber style integration tests and then very quickly switched over to property based tests. Making that shift from traditional testing to property-based testing benefited us because we found certain bugs that we would not have found out with regular unit testing.
Can you give me an example of one of the stories you will tell in the talk?
When I first started using property-based testing, it was on OSCC (or Open Source Car Control platform). OSCC is a lot of Arduino firmware and stuff like that. We had multiple unit tests written for it but we were pretty much considering typical usage. With the property based tests, we found overflow errors or casting errors where we were storing to Int32s. For example, if you passed a negative number it would underflow. Since we were not considering these conditions, it pretty much resulted in an overhaul of a bunch of that code when we realized that a lot of the internal stuff was not safe as we thought it was.
How much of the talk is about git in this metadata vulnerability versus property-based testing? Is it about both or is it about property-based testing?
It is about both. To introduce attendees to the benefits of property-based testing, I will be using examples from our work both on supplementing our unit tests for OSCC and creating system tests for the git metadata solution.
What do you feel is the most important trend in software right now?
I think the most important trend is going to be shifting away from the importance of AI and algorithms and shifting to a focus on safety - making sure everything works correctly because it’s important to feel confident that software is correct before putting it into a vehicle and out on the road. What I am trying to say is I think automation is really important. I think it is going to make a lot of things safer and a lot of things easier.
Similar Talks
Tracks
-
Microservices: Patterns & Practices
Evolving, observing, persisting, and building modern microservices
-
Developer Experience: Level up Your Engineering Effectiveness
Improving the end to end developer experience - design, dev, test, deploy, operate/understand. Tools, techniques, and trends.
-
Modern Java Reloaded
Modern, Modular, fast, and effective Java. Pushing the boundaries of JDK 9 and beyond.
-
Modern User Interfaces: Screens and Beyond
Zero UI, voice, mobile: Interfaces pushing the boundary of what we consider to be the interface
-
Practical Machine Learning
Applied machine learning lessons for SWEs, including tech around TensorFlow, TPUs, Keras, Caffe, & more
-
Ethics in Computing
Inclusive technology, Ethics and politics of technology. Considering bias. Societal relationship with tech. Also the privacy problems we have today (e.g., GDPR, right to be forgotten)
-
Architectures You've Always Wondered About
Next-gen architectures from the most admired companies in software, such as Netflix, Google, Facebook, Twitter, Goldman Sachs
-
Modern CS in the Real World
Thoughts pushing software forward, including consensus, CRDT's, formal methods, & probalistic programming
-
Container and Orchestration Platforms in Action
Runtime containers, libraries, and services that power microservices
-
Finding the Serverless Sweetspot
Stories about the pains and gains from migrating to Serverless.
-
Chaos, Complexity, and Resilience
Lessons building resilient systems and the war stories that drove their adoption
-
Real World Security
Practical lessons building, maintaining, and deploying secure systems
-
Blockchain Enabled
Exploring Smart contracts, oracles, sidechains, and what can/cannot be done with blockchain today.
-
21st Century Languages
Lessons learned from languages like Rust, Go-lang, Swift, Kotlin, and more.
-
Empowered Teams
Safely running inclusive teams that are autonomous and self-correcting