Responding to you and not the submitter because I'm not sure what they mean either:

An AI that generates binaries directly is a compiler. Or a compiler/assembler/linker in one, whatever.

So it raises the question of what language humans should use to communicate with this compiler. What language does it compile?

If the language is some existing programming language, then how can the "AI" technology help much? Suppose it compiles C. The semantics of the existing language have to be maintained or it isn't C. So maybe AI can improve optimization by 0.0001%. Or do something really ridiculous based on the undefined behavior loophole. Who cares?

English (or Chinese or Norwegian) are not programming languages - they are too ambiguous.

Therefore it makes sense to me that a higher-level programming language than existing ones would be useful to talk to this hypothetical compiler - a language that gives the AI more freedom than the existing languages, but more precise than languages used by humans to talk to each other.

Maybe someone already has invented one, but probably the last word hasn't been written on the subject yet.

> Why not train the AI on machine language and have it generate binaries directly?

Maybe that's the answer I was looking for, AI can "interpret" sentences referring to the problems/needs to be solved and it may lack the way to solve for those with the existing programming languages and it could come up with it's own "abstraction" on top of processes - or maybe not, and will plainly speak low-level 0-1s directly.

And AI is a mind-reader that can tell itself what binaries you want it to produce?

"Let's see now. That human obviously wants me to produce a binary for a self-driving car... or perhaps it was a binary for a game of Chess ... or perhaps it was a binary for a new version of Windows .... or perhaps .... Well, I don't know!"