New Delhi: Days after OpenAI launched a new model Astra, the company’s chief scientist, Jakub Pachocki has sounded a warning about the future of AI.
“This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. OpenAI will continue to seek technical solutions to alignment and monitoring, to build defensive systems and unilaterally withhold further scaling as needed; however, I believe broader interventions are required,” Pachocki wrote in an essay published on 6 September.
In the essay, titled “An Alien Mind”, he writes that autonomous AI models “are becoming superhuman in their ability to break in and out of computer systems”. According to him, this expands the risks associated with AI “tremendously” putting humans in a “narrow window” where we must “tighten security of critical systems”.
“The core challenge of automating AI research is not “getting there” – it is getting there in a way that keeps people a part of the continued improvement process, and leaves the future in humanity’s hands,” he wrote.
Also read: Two keys, one dilemma—the story of Modi’s slum neighbours evicted to new homes 50 km away
Can we really ‘understand’ AI?
He explained that AI is not “designed” and is instead “grown”. It is the byproduct of repeating a series of optimisations and the result is a complex system that can simulate aspects of human behaviour. According to Pachocki, although we can make sense of some of its mechanisms, the entirety of it eludes human understanding — much like scientists the way scientists grapple with neuroscience itself.
“We put a lot of effort into building principled algorithms and making testable predictions, but fundamentally, our large-scale training runs are experiments, and we are sometimes surprised by their results. Moreover, as the systems become more capable, the results become harder to interpret,” he added.
In the future, Pachocki says that AI will become a key aspect of its own development. Even now, the term “recursive self improvement” means that AI is already involved in AI research. Such kind of automated research could speed things up exponentially, but Pachocki says that accelerating deep learning is not the “right collective action we should take as a research community”.
However, Pachocki says that the kind of intelligence AI develops through deep learning is not comparable to human intelligence. What makes this particularly alarming is that machine intelligence and human intelligence are born out of different processes, and scientists cannot assume that AI will adhere to human principles by default.
“The core problem in AI research is that of alignment – getting the AI to “try to do the right thing” by human standards… Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision,” said Pachocki.
As governments across the world are trying to find ways of regulatory frameworks around AI, Pachocki says that no lab has figured out how to monitor AI enough to continue scaling at an uncontrollable pace.
“I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established,” he said.