> Fuzzing would be unlikely to discover a bug that only occurs on giant inputs or needs a special configuration of lists.
I have a concern about peoples' over confidence in fuzz testing.
It's a great tool, sure, but all it is is something that selects (and tries) inputs at random from the set of all possible inputs that can be generated for the API.
For a strongly typed system that means randomly selecting ints from all the possible ints for an API that only accepts ints.
If the API accepts any group of bytes possible, fuzz testing is going to randomly generate groups of bytes to try.
The only advantage this has over other forms of testing is that it's not constrained by people thinking "Oh these are the likely inputs to deal with"
This is not quite true, what you are describing is "dumb" fuzzing. Modern fuzzers are coverage guided and will search for and devote more effort to inputs which trigger new branches / paths.
I have a concern about peoples' over confidence in fuzz testing.
It's a great tool, sure, but all it is is something that selects (and tries) inputs at random from the set of all possible inputs that can be generated for the API.
For a strongly typed system that means randomly selecting ints from all the possible ints for an API that only accepts ints.
If the API accepts any group of bytes possible, fuzz testing is going to randomly generate groups of bytes to try.
The only advantage this has over other forms of testing is that it's not constrained by people thinking "Oh these are the likely inputs to deal with"