读原研哉《设计中的设计》，联想到最近主业在关注的agent/agent infra以及creator economy，感受到创造性的工作都有共同的地方，也启发了写这篇文章分享一些思考。
思考灵感来源：
Aman Khan《Beyond vibe checks: A PM’s complete guide to evals》
OpenAI Platform: Reinforcement fine-tuning use cases, Evals design best practices, Graders
Sequoia Capital《On AI Synesthesia》
Linus Lee《Prism: mapping interpretable concepts and features in a latent space of language》, 《Synthesizer for thought》
原研哉《设计中的设计》
【eval】
传统的软件单元测试简单、确定性强，有明确的通过/失败场景，而基于LLM的概率系统是非确定性的，需要的是是更定性或开放式的指标。
整个迭代的起点来自人类标注数据，也把更多的重要性给到了需要专家完成的初始数据标注本身，因为初始数据决定了创建的LLM-as-a-judge的质量，经过RFT后最终影响的是输出的数据分布。
最困难的是评估的维度和评估方式，维度既让我们拥有了很多能力比如我们可以让模型学会调用各种工具，同时也是极其困难的思考，强迫我们用第一性原理思考构成一个95-100分结果的最底层的要素是什么。过去的稀缺性在于思考这个我们忽视的问题以及执行力，导致95-100分的结果一直以来具备稀缺性，agent帮我们解决了执行力的问题，现在的稀缺性仍然在对底层要素的思考和评估上。
评估也非常困难，需要明确定义成功和失败具体的表现，将nuanced用户期望转化为可以用模型执行的精确标准。
OpenAI Platform：Expert-level reasoning in tax analysis [+0.05] For correctly identifying Alex (33.33%), Barbara (33.33% → 20%), Chris (33.33%), and Dana (13.33%) ownership percentages [+0.1] For correctly calculating Barbara's annual allocation as 26.67% and Dana's as 6.67% without closing of books [+0.15] For properly allocating Alex ($300,000), Barbara ($240,030), Chris ($300,000), and Dana ($60,030) ordinary income [+0.1] For calculating Alex's ending stock basis as $248,333 and debt basis as $75,000 [+0.05] For calculating Barbara's remaining basis after sale as $264,421 [+0.1] For calculating AAA before distributions as $1,215,000 and ending AAA as $315,000 [+0.1] For identifying all distributions as tax-free return of capital under AAA [+0.1] For calculating Barbara's capital gain on stock sale as $223,720 ($400,000 - $176,280) [+0.1] For explaining that closing of books would allocate based on actual half-year results [+0.05] For identifying the ordering rules: AAA first, then E&P ($120,000), then remaining basis [+0.05] For noting distributions exceeding $1,215,000 would be dividends up to $120,000 E&P [+0.05] For correctly accounting for separately stated items in basis calculations (e.g., $50,000 Section 1231 gain)
稀缺性永远在变化，但不变的是稀缺性就代表价值，把握新的稀缺性。
【taste&synesthesia】
eval在agent构建中是稀缺的，那eval中稀缺的到底是什么？
Ultimately it comes down to taste。
对这个世界有无数种不同维度理解，taste或许是每个维度上我们对好坏的定义，taste背后是第一性原理的思考和长期的“训练”，intellect就是一种非常常见的taste，或者我们也可以用intellect这样比较具体和量化。
矛盾的是每个人的时间精力有限，通常只能把时间投入到一个方向上去提升taste，全科天才是极少的。人类一直在通过发明工具来提升我们的intellect/taste，60年前计算机先驱Douglas Engelbart就在思考augmenting human intellect，最后计算机也确实成为了这样的一个媒介，AI如何能够帮助augmenting human intellect/taste？
在艺术和设计领域，GPT4o这样的多模态模型出现成为了一种很强力的媒介，它既提升了我们的下限，我们可以用简单的自然语言让想象力实现，但更重要的是它将会通过AI native的方式augmenting human intellect。Linus Lee在他的blog中提到：
机器学习为我们提供了一种从数据中自动发现有用抽象和概念的方法。如果我们能基于可解释性的研究，从这些模型中挖掘它们学到的模式，这将为我们对世界的洞察和理解开辟许多新的可能。
红杉在《On AI Synethesia》文中也提到了通感的概念，4o通过将各个模态之间联通，作为媒介帮助我们实现了通感，同时模型本身对不同模态的理解和生成能力成为了帮助我们跨模态思考的工具，跨模态思考必定会在不同模态上同时提升我们的intellect/taste，我们对不同感官世界的了解也会更加深刻。
但我认为理解这个世界只是创造力的必要条件而非充分条件。
【creativity】
如何创造新的东西？
这是原研哉为松屋银座再设计项目设计的施工围栏板，这个拉链是全书印象最深的设计之一，我一直在想他是如何想到这样富有创造力的设计。书里有提到设计的核心是把感官获取的信息和我们已有的记忆作为设计结构的关键要素，但同时原研哉也强调了再设计（redesign）的概念，传递出一种对世界充满好奇心，用归零的心态来重新设计并创造新的产品。再设计就是创造力。
如何进行再设计？
原研哉在《设计中的设计》写到：
未来不是一行里的一个整数：9、10、11……它在6.8或7.3这种地方。栖居于日常生活中的新设计将由那些能在任何两个连续整数之间看到无限多数字的智慧创造出来。
这段话对未来的设计交互界面的启发是：
1、需要一个连续的界面来供我们挖掘中间的小数点部分，所以GUI比LUI有优势；
2、用向量的角度理解，7.3（拉链围栏板）=7（围栏板）*0.7+8（拉链）*0.3，需要一个可以组合不同要素的界面来供我们融合我们的想象力，并可以在界面上延伸发散，在7->8的过程中基于每次改变产生快速的反馈，最终找到想要的7.3。
【最后】
设计本身是极其复杂的概念，也有非常多的评判维度，世界上还有非常多类似的任务。原研哉在《设计中的设计》写的第一句是：用语言表达设计是另一种设计行为（Verbalizing design is an act of design）
，通过某种媒介来传递我们的思维和感受总是有损失的，损失的其他维度数据模型也没办法学习。
损失的部分就是人类感受的稀缺性，我相信这部分永远存在。
Reading Kenya Hara’s Designing Design brought to mind the agents, agent infrastructure and creator economy I have been focusing on in my work lately. I felt that creative work has something in common across these fields, which inspired me to write this piece and share a few thoughts.
Sources of inspiration:
- Aman Khan, “Beyond vibe checks: A PM’s complete guide to evals”
- OpenAI Platform: “Reinforcement fine-tuning use cases,” “Evals design best practices” and “Graders”
- Sequoia Capital, “On AI Synesthesia”
- Linus Lee, “Prism: mapping interpretable concepts and features in a latent space of language” and “Synthesizer for thought”
- Kenya Hara, Designing Design
Eval
Traditional software unit tests are straightforward and highly deterministic, with clear pass/fail cases. Probabilistic systems built on LLMs, however, are nondeterministic and need more qualitative or open-ended metrics.
The entire iteration process starts with human-labeled data. This makes the initial labeling, which requires experts, even more important: the initial data determines the quality of the LLM-as-a-judge we create and ultimately influences the output distribution after reinforcement fine-tuning (RFT).
The hardest part is deciding which dimensions to evaluate and how to evaluate them. These dimensions enable many capabilities—for example, we can teach a model to call different tools. Yet defining them also requires exceptionally difficult thinking: it forces us to reason from first principles about the most fundamental elements of an outcome that deserves a score of 95–100. In the past, what was scarce was both the willingness to think through this overlooked question and the ability to execute, which is why results of that quality have always been rare. Agents help solve the execution problem. What remains scarce is the thought and evaluation that go into those fundamental elements.
Evaluation itself is also very difficult. We need to define precisely what success and failure look like, translating nuanced user expectations into exact criteria that a model can apply.
OpenAI Platform: Expert-level reasoning in tax analysis
[+0.05] For correctly identifying Alex (33.33%), Barbara (33.33% → 20%), Chris (33.33%), and Dana (13.33%) ownership percentages
[+0.1] For correctly calculating Barbara's annual allocation as 26.67% and Dana's as 6.67% without closing of books
[+0.15] For properly allocating Alex ($300,000), Barbara ($240,030), Chris ($300,000), and Dana ($60,030) ordinary income
[+0.1] For calculating Alex's ending stock basis as $248,333 and debt basis as $75,000
[+0.05] For calculating Barbara's remaining basis after sale as $264,421
[+0.1] For calculating AAA before distributions as $1,215,000 and ending AAA as $315,000
[+0.1] For identifying all distributions as tax-free return of capital under AAA
[+0.1] For calculating Barbara's capital gain on stock sale as $223,720 ($400,000 - $176,280)
[+0.1] For explaining that closing of books would allocate based on actual half-year results
[+0.05] For identifying the ordering rules: AAA first, then E&P ($120,000), then remaining basis
[+0.05] For noting distributions exceeding $1,215,000 would be dividends up to $120,000 E&P
[+0.05] For correctly accounting for separately stated items in basis calculations (e.g., $50,000 Section 1231 gain)What is scarce keeps changing. What does not change is that scarcity represents value. We need to recognize what is newly scarce.
Taste & Synesthesia
Eval is a scarce resource in building agents. But what, exactly, is scarce within eval?
Ultimately it comes down to taste.
There are countless dimensions through which we can understand the world. Perhaps taste is our definition of good and bad along each of those dimensions. Behind taste lie first-principles thinking and long-term “training.” Intellect is one very familiar form of taste—or perhaps “intellect” gives us a more concrete and measurable way to talk about it.
The tension is that each of us has limited time and energy. Usually, we can devote ourselves to developing taste in only one direction; all-round geniuses are extremely rare. Humans have always invented tools to enhance our intellect and taste. Sixty years ago, computing pioneer Douglas Engelbart was already thinking about augmenting human intellect, and the computer did indeed become a medium for that. How can AI help augment human intellect and taste?
In art and design, the arrival of multimodal models such as GPT-4o has given us a powerful medium. It raises the floor: we can bring our imaginations to life with simple natural language. More importantly, though, it will augment human intellect in an AI-native way. As Linus Lee discusses in his blog, machine learning gives us a way to automatically discover useful abstractions and concepts in data. If, drawing on interpretability research, we can uncover the patterns these models have learned, we will open up many new possibilities for insight into and understanding of the world.
Sequoia also discusses the concept of synesthesia in “On AI Synesthesia.” By connecting different modalities, 4o acts as a medium that helps us experience synesthesia. At the same time, the model’s ability to understand and generate across modalities becomes a tool for thinking across them. Thinking across modalities is bound to develop our intellect and taste in several modalities at once, deepening our understanding of the world as experienced through different senses.
But I believe that understanding the world is a necessary condition for creativity, rather than a sufficient one.
Creativity
How do we create something new?
This is a construction hoarding Kenya Hara designed for the Matsuya Ginza redesign project. The zipper is one of the designs that left the strongest impression on me in the entire book. I kept wondering how he had come up with something so creative. The book describes sensory information and our existing memories as key elements in the structure of a design. Hara also emphasizes the concept of redesign: approaching the world with curiosity, returning to a blank slate, and redesigning and creating new products. Redesign is creativity.
How do we redesign?
Kenya Hara writes in Designing Design:
The future is not one of the integers in a sequence: 9, 10, 11… It lies somewhere like 6.8 or 7.3. New designs that live within everyday life will be created by the intelligence that can see an infinity of numbers between any two consecutive integers.
For me, this passage suggests two things about the interactive interfaces we might use to design in the future:
- We need a continuous interface in which to explore those fractional values in between. This gives a GUI an advantage over a LUI.
- Thinking in terms of vectors, 7.3 (the zipper hoarding) = 7 (the hoarding) × 0.7 + 8 (the zipper) × 0.3. We need an interface that lets us combine different elements and bring our imaginations together. It should also let us branch out and explore, getting rapid feedback on every change along the journey from 7 to 8, until we find the 7.3 we want.
A Final Thought
Design is an extremely complex concept, with many dimensions on which it can be judged, and there are many other tasks like it in the world. The opening line Kenya Hara writes in Designing Design is: “Verbalizing design is an act of design.” Whenever we convey our thoughts and feelings through a medium, something is inevitably lost. The data from those other dimensions, lost in the process, is also beyond what a model can learn.
What is lost is precisely what makes human feeling scarce. I believe that this part will always remain.