If you haven't played with AFL I recommend spinning it up for a night and trying it out.
My experience was pretty fun. The setup wasn't that difficult if you RTFM. I rebuilt OpenSSL from source the way AFL wanted and started throwing test cases at asn1parse. The test cases it was coming up with were really cool. Got one or two hits in my hour or two running it but they ended up being false positives.
For a side project, there is really just a ton of work that has gone into it. lcamtuf is doing a great job.
Hm, is the use of the preeny library really necessary to get nginx reading from stdin? Couldn't you just 'cat' your fuzzed input files to 'nc' which then connects to the socket nginx is listening to?
How important is performance for an offline testing tool, when compared to flexibility? How many bugs will mucking with intercepting syscalls introduce? More importantly, how many bugs will be obscured since the tool is running Nginx in a decidedly non-standard and complex way?
How important is performance for an offline testing tool, when compared to flexibility?
Given that this specific form of fuzzing - basically non-directed randomization of inputs (it is pattern based to an extent, but...) performance matters. You want to cover a good portion of a huge surface. Performance tweaks add up in this case.
How many bugs will mucking with intercepting syscalls introduce? More importantly, how many bugs will be obscured since the tool is running Nginx in a decidedly non-standard and complex way?
Good questions - but in the overall workflow, these can be easily found by first running the fuzzing and getting a list of problematic inputs, and second, trying those inputs in a directed way against the unmodified version of nginx provides a nice filter.
As for the obscuring of other bugs - sure, this is always a problem with any form of testing. The goal with this or any tool isn't to find all bugs, its to help find a class of bugs. Other forms of testing will find other bugs. This is why unit tests are valuable, but don't obviate the need for integration and functional testing for example.
Suggestion: You could test multiple requests by feeding nginx a test input that consists of several concatenated HTTP requests. Since you're using HTTP/1.1 and the default response is keep-alive, the multiple requests will get pipelined together.
Would it be possible to use this method against something like a Java app loaded in tomcat where as to test for bugs within a certain library? Say, for instance that I wanted to see if certain malformed xml posts were able to cause unexpected behaviors in a passing endpoint? As I write this I think that in some cases some kind of httppenetration tool might be more suited but I think im wondering what would happen with a tool that isn't necessarily confined to a ruleset and pattern matching.
The afl fuzzer relies on compiling C code with its own compiler, so I think it's limited to only C based programs.
What you may want is to use something like `quickcheck` (scalacheck or clojure's test.check I guess?) to send lots of "arbitrary" xml at your code and see what breaks. With sufficiently interesting definitions of "arbitrary" you can probably find bugs.
That approach would be testing inside the process, as opposed to passing in whole http requests. But if you know a section of code is more vulnerable than others, focus on it. No need to test all of tomcat's http parsing when you really care about your specific library.
I think you meant exits instead of exists.
Nice write up btw.