I use artificial intelligence (AI) daily, but mostly as a sophisticated googling tool. For example, my recent chats include (as names given by ChatGPT): matrix plot visualisation, image registration issues (of course, who doesn’t have them), change tax class in Germany (navigating German bureaucracy), soy in white bread, and Python file copy script. Based on my experience, the performance of AI is far from perfect, and often the answer is plainly wrong. However, from every vacuum cleaner, I keep hearing that it’s very soon we will have superintelligent AI. To be somewhat prepared, I decided to read a book “Superintelligence: Paths, Dangers, Strategies” by Nick Bostrom.

The book was written in 2014, but most of it is relevant to current trends. It seems that discussions about the future with superintelligence were rather rare in 2014, so this book was a starting point.

The author begins by discussing possible routes for superintelligence. Superintelligent AI is only one of them. The possible routes are:

1. Starting from scratch and simulating the whole evolution from the beginning, and then continuing to simulate, and hoping to reach superintelligence

2. Come up with an abstract model of a human brain (like a predictive processing framework) and simulate, and hope to reach superintelligence

3. Create child-like AI or, as the author calls it, seed AI, and hope that it will improve itself substantially, that it will grow into superintelligence

4. Simulate the whole brain in detail, all the molecules and all the cells, and hope to reach superintelligence; the author terms it whole brain emulation

5. Attempt to create a superintelligent human by selective breeding with additional use of genetic editing

6. Enhance contemporary humans with brain-computer interfaces (BCI) to the point of superintelligent beings

7. Create a web-based cognitive entity using accumulated knowledge from the Internet (wait, it’s ChatGPT, are we there yet?)

Some paths are more probable than others. For instance, selective breeding is ethically very questionable, while scraping websites is not. Regardless of the path taken, the author claims that superintelligence will have the same instrumental values, namely:

1. If a superintelligent agent has a goal, it will want to survive (if you are dead, then you didn't fulfill your purpose). Hence, self-preservation would characterize every superintelligence.

2. The agent will pursue their goals, which are supposed to stay constant over time. Hence, every superintelligent agent will have goal-content integrity.

3. If an agent has a final goal, it would be beneficial for them (and they will understand that) to become cleverer to finally attain a goal. Hence, every superintelligence will pursue cognitive enhancement.

4. Along the same lines, having better hardware would enable an agent to achieve the goal more quickly. Hence, the agent will go after technological perfection.

5. Technological perfection will require materials (more data centers), supply (electricity, water), or support (from other minds). Hence, superintelligent agents will deploy resource acquisition scenarios.

The author calls it “the instrumental convergence thesis” - regardless of the type of superintelligence, it will possess those instrumental values. If we accept the instrumental convergence thesis and the instrumental values, it becomes apparent that this superintelligence may try to take over the world. Keeping superintelligence from taking over the world is a control problem. The author classifies control methods into capability control and motivation selection. Examples of capability control are boxing (no internet access), incentivizing (rewards for being “good”), stunting (limiting cognitive capacities or hardware), and tripwires (extensive diagnostic tests for violations). Motivation selection is about setting a set of rules or norms or ways to arrive at the desired (by the developers) rules and norms. Obviously, all control methods are not mutually exclusive and can (and should) be applied together.

If there is a possibility that superintelligence will take over the world, why do we need it? The author sees superintelligence as a means for endless possibilities: treating diseases, resolving climate change, colonizing other planets, and answering all philosophical questions, among many others.

To achieve this bright future, the author suggest cooperations. The competition between developers, according to the author, is detrimental to progress and contains more risks than benefits. Therefore, businesses and countries that work on developing superintelligence should join forces.

The book is clear and easy to read. The author points out that it is important to have a discussion about the way humanity proceeds sooner rather than later, and with that, I agree. However, the book also contains way too much scaremongering for my taste. There are also inconsistencies. For instance, the author makes an example of how setting the goal in an inaccurate way may lead to disaster. If AI is asked to produce exactly 1 million paper clips, it may end up counting paperclips over and over again to make sure it’s correct. It is hard to believe that we may call this AI a superintelligence.  Also, instrumental values (that the author believes any kind of superintelligence will possess) all include some form of self-referential “thought”. It is assumed that the agent will refer to a goal as “my goal”. This is not obvious at all when it comes to artificial systems.

I felt mildly desolate while reading the chapter about values, namely, what kind of values we should embed into superintelligent AI. I have my own values, but other people may not share them. So my values are not universal, but how can we arrive at universal values? The ways that the author offers are vague and obscure, but I don’t have better ideas, which made me feel uncomfortable. 

I cannot recall the state of neuroscience and philosophy in 2014, but as of 2026, it would be very nice to have tools to evaluate the system for superintelligent qualities (while evrybody are trying to figure out if ChatGPT has consciousness, maybe we should try to figure out if it is superintelligent). The author provides the definition that superintelligence is the entitity who’s intelligence exceeds that of all humanity combined. ChatGPT can definitely pass at least some intelligent tests better than anyone on this planet, but does it make ChatGPT a superintelligence? It would be nice to understand what superintelligence exactly is, so as to know if you encounter one.

Favorite quote:

“The common good principle:

Superintelligence should be developed only for the benefit of all of humanity and in service of widely shared ethical ideas”

January, 2026
Written by a human