Edward Boyden has a fascinating essay at MIT’s Technology Review website in which he describes a problem that could possibly arise from super-smart artificial intelligence. The problem, Boyden notes, is motivation: even with all of that intelligence and computational, how does a possibly sentient computer become moved to utilize that power?
...Indeed, a really advanced intelligence, improperly motivated, might realize the impermanence of all things, calculate that the sun will burn out in a few billion years, and decide to play video games for the remainder of its existence, concluding that inventing an even smarter machine is pointless. (A corollary of this thinking might explain why we haven’t found extraterrestrial life yet: intelligences on the cusp of achieving interstellar travel might be prone to thinking that with the galaxies boiling away in just 1019 years, it might be better just to stay home and watch TV.) Thus, if one is