It sounds like you missed the x11vnc part of this. You connect into the container using vnc, not x11, and potentially via a browser based vnc client. If you want to see a working example of a gui app running successfully in docker, check out https://github.com/jlesage/docker-handbrake