George Dvorsky has a piece on iO9, How Artificial Intelligence Will Give Birth to Itself, summarizing many of the worrisome aspects of a possible runaway effect, when self-improving artificial intelligences (AI) get faster and faster at designing new and better versions of themselves. A thoughtful reflection on how the Singularity might (or might not) go out of control.

Alas, George left out a process issue that makes all the difference. That issue is Secrecy, which lies at the root of every Michael Crichton science-goes-wrong scenario. (Not one of Michael’s plot drivers would have taken place, if the “arrogant scientists” had done their innovating in the open - as most scientists have been trained to prefer - exposing their new robots/dinosaurs and so on to truly public, error-correcting criticism.)

Efforts to develop AI that are subject to the enlightenment process of reciprocal scrutiny might see their failure modes revealed and corrected in time. Those that take place in secret are almost one hundred percent guaranteed to produced unexpected outcomes. And most likely dangerous ones.

More: CONTRARY BRIN: Science That Threatens… and Promises Wonders