The “two kids in a trenchcoat” trope, in case you haven’t heard of it. Image sources: 1, 2tl;drI go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So it's bad to lump them together and call them by the same name.1. An example “coherence argument” that I’ll focus on for most of the postI’ll talk about two other “coherence arguments” at the end (§6), but let’s start with the one based on market competition:In a stock market, people with circular preferences will systematically transfer money to those without circular preferences. For example, if I think I’m winning every time I trad...