I'm not sure that's true. For example, Supernote have done very well with their A6X/Nomad range (confusingly about the size of an A5 sheet of paper), I think partly because they're the only company that has targeted that size. Personally, I much prefer having a smaller page to make notes on, and that's part of the reason I went for the Supernote.

If you're reading and annotating a lot of full-sized PDFs, then bigger is better. But for just taking notes, drawing diagrams, etc, I found a smaller device was a lot more portable and convenient.