# Ratcheting progress in tools for thought ## Metadata - Author: [[Andy Matuschak]] - Full Title: Ratcheting progress in tools for thought - Category: #articles - URL: https://www.patreon.com/posts/ratcheting-in-47976114 - My highlights: https://readwise.io/reader/shared/01gqmvk9nhdpfcamerjfkt34dj/ ## Highlights The difference is that a field is about ratcheting: developing a growing shared corpus of general knowledge and methods which allow projects to meaningfully build on each other, across researchers and across years, on and on in an upward cycle. ([View Highlight](https://read.readwise.io/read/01h6yanmk5c2nkav0hqfhe453b)) An idealized cycle of activity might involve something like these key steps: 1. **Identifying powerful insights** about some subject domain or about cognition in general which might be fruitfully systematized 2. **Building systems** which express those insights in their primitives 3. **Observing serious use** of those systems in authentic contexts, and of your theoretical insights refracted through them 4. **Distilling generalizable insight** from those observations which produce new understanding about the subject domain or cognition, and which permit new, better systems to be built 5. **Disseminating** that insight so that others can build on it ([View Highlight](https://read.readwise.io/read/01h6yapytkn27tdrf7w4wqrzb0)) Perhaps the most common pattern of all is that people in the tech industry focus mostly on building systems. Those systems are usually expressions of technological or market insights, rather than of fundamental insights about a subject domain or about cognition. ([View Highlight](https://read.readwise.io/read/01h6yaxv5bwfcwzzrgszgc3jfn)) I think good tools-for-thought research often focuses on transcending and discarding the current system, asking “how should we build the next system?”. But good business usually don’t throw out their core product and build a meaningfully different one every few years. ([View Highlight](https://read.readwise.io/read/01h6yb03jstcfdernc23cx0awf)) The most common pattern seems to be: a bricoleur identifies some powerful idea about a representation and designs a prototype, but then fails to engage seriously with observing and deriving insight from the systems they’ve built. Sometimes this comes from technical barriers—the prototype is too quick-and-dirty to be used in a serious context, so their insight is limited. But I think [there’s also a cultural gap here](https://andymatuschak.org/2020/), a missing research practice of careful, diligent observation and synthesis. Too often these projects have the flavor of “Look, I made a thing! Isn’t it cool? How many people can I get to use it?”. But the question they need to be answering is: “What powerful, generalizable ideas can we learn from this project? How should the next wave of systems build on this?” ([View Highlight](https://read.readwise.io/read/01h6yb1ym9cvtpfmxe94g7qgvx)) But what should the field learn from this experiment? What generalizable conclusions can we draw? How can we improve our understanding of that initial idea so that we can create a better future medium? ([View Highlight](https://read.readwise.io/read/01h6yb611yjtz8gkgaf6d599d0)) - Note: Good prompt questions for discerning generalizable insights with a research prototype. Somewhat more subtly, I’ve found that designing tools-for-thought experiments such that the results might tell you something possibly *generalizable* is particularly difficult—a contorted balancing act of theory, interface design, engineering, and experimental methods. ([View Highlight](https://read.readwise.io/read/01h6yb96g9ysgm3x99ne3vkbbb)) The key thing it does is to explicitly connect the dots between a grounded theoretical claim, the implied design approach, and the desired outcome. ([View Highlight](https://read.readwise.io/read/01h6ybbnk3yptzchrhhjc9075g)) Or, to take another example, I know many of my readers are fans of outline-based text editing. This morning, inspired by a message from patron Ethan Plante, I went looking for academic work on the theoretical or empirical foundations of outline processors. I was shocked how little I could find. So, if you’re experimenting with building outline processors, or “block-based” tools, or whatever, some questions to be answered: what effects do these alternative writing primitives have on composition? on thinking? on reading? What is the theory which would explain these effects? How would we know if it were true? What else does that theory imply? What would research systems which could answer these questions look like, and how are those different from the commercial systems people build? ([View Highlight](https://read.readwise.io/read/01h6ybfqccg68knmjynxnrbn54)) ## New highlights added August 10, 2023 at 10:42 AM businesses are trying to improve their product’s performance in its customers and market. Sometimes we get lucky, and this happens to produce generalizable insights—typically only after others reverse-engineer and disseminate them. But often this difference in focus means a narrower scope, as far as the field is concerned. ([View Highlight](https://read.readwise.io/read/01h7fwak2vzcd1a59zjexdy4dz)) the system has to be shaped in a way which allows you to ask the questions you want to ask. But often you can’t even identify the right questions to ask before you see the system in operation! ([View Highlight](https://read.readwise.io/read/01h7fwch7hq6ss6qq52r2nhe7k)) - Note: How does one build systems designed for the asking and answering of questions? [Educational design] principles are usually heuristic statements of a format such as: “If you want to design intervention X (for the purpose/function Y in context Z), then you are best advised to give that intervention the characteristics A, B, and C (substantive emphasis), and to do that via procedures K, L, and M (procedural emphasis), because of arguments P, Q, and R [(theoretical emphasis)].” ([View Highlight](https://read.readwise.io/read/01h7fxf692egjf70dcyhr340nj)) What are the consequences of our theories and our design decisions, the consequences which others can build on? ([View Highlight](https://read.readwise.io/read/01h7fy4qxp8k53bajtt5vmqxdh)) - Note: good prompt