But a derivative work requires that major copyrightable elements remain. So it seems like there is a fuzzy line on whether something is really derivative or not.
I don't think it is necessarily enough to have simply started with a copyrighted work.
Yes, it has to be visually derivative in order to be a copyright infringement, not mathematically derivative!
Imagine if Stable Diffusion was made illegal. Someone accuses me of using this illegal tool for an image that doesn’t look like anyone else’s image as far as the court is concerned for copyright. I put the image on my website. If the image itself is not at all infringing, then what is the evidence that Stable Diffusion was used? Should the police be issued a warrant to search my private property for proof that I used Stable Diffusion without a shred of evidence?
I don't think it is necessarily enough to have simply started with a copyrighted work.