I don’t think the claim was that it had no recoil; rather, I believe it was designed to fire bursts so fast that it could send three rounds downrange before the operator perceived the recoil, thus improving accuracy.
Beyond that: I don’t really know. If I had to guess, I’d say the G11 is an example of one of those guns that comes along once in awhile that’s basically the solution to problems nobody really has. The idea of a rifle that can easily carry 50 rounds, and is immune to the sorts of malfunctions/stoppages that can be associated with metal casings, is intriguing…but not really the answer to anyone’s prayers.
Moreover: forward leaps in technology with no proven track record can be scary and intimidating to organizations in any context, especially when the industry is one in which reliable and effective performance is literally the difference between “life” and “death.” It may seem funny now, but when the first M16 rifles made it to line troops during the Vietnam War, one of the reasons why it was initially regarded with distrust was that troops accustomed to large-bore weapons of forged iron/steel and wood felt that the new space-age weapon made of lightweight composite alloys and polymers was flimsy and toy-like (“the Mattel plastic gun”) and assumed it’s small, high-velocity bullet wouldn’t inflict much injury.
H&K’s M8 rifle, which in my opinion incorporated a lot of great features— particularly it’s highly-modular nature— has nonetheless failed to get much traction (or buyers) despite getting serious consideration in military trials by a number of countries, including the US.
Other typical reasons for certain weapons not getting wide acceptance: politics, military bureaucracy, cost of purchase/integration of new weapons systems.