Source: Quantitative Investment & Machine Learning, 2023-05-24

“From a computing power perspective alone, 幻方 (High-Flyer) may have even obtained the ticket to build a ChatGPT-like model earlier than many large tech companies.”

Amid the frenzy of companies racing to build large language models, High-Flyer (幻方) is perhaps the most unconventional player.

This is a game destined for the few. Many startups began to pivot or even consider withdrawing after large tech companies entered the field. Yet this quantitative hedge fund pressed forward relentlessly.

In May, High-Flyer named its independent new organization for building large models “DeepSeek” (深度求索), emphasizing its focus on creating truly human-level artificial intelligence. Their goal is not merely to replicate ChatGPT, but to explore and unravel more mysteries surrounding Artificial General Intelligence (AGI).

Moreover, in a track that is widely considered to heavily rely on scarce talent, High-Flyer seeks to assemble a group of driven individuals, wielding what they consider their greatest weapon: collective curiosity.

In the quantitative field, High-Flyer is a “top-tier fund” that has reached the scale of 100 billion RMB. Yet the way it captured the spotlight in this AI wave was rather dramatic.

When shortages of high-performance GPU chips from domestic cloud providers became the most direct factor limiting the birth of generative AI in China, it was reported by Caijing Eleven that no more than five enterprises in China possessed more than 10,000 GPUs. Besides several leading tech giants, one was a quantitative hedge fund named High-Flyer. It is generally believed that 10,000 Nvidia A100 chips represent the computing threshold for training self-owned large models.

In fact, this company, rarely viewed through the lens of artificial intelligence, has long been a hidden AI giant: In 2019, High-Flyer established an AI company. Its self-developed deep learning training platform, “Firefly No. 1,” invested nearly 200 million RMB and was equipped with 1,100 GPUs. Two years later, the investment in “Firefly No. 2” increased to 1 billion RMB, equipped with approximately 10,000 Nvidia A100 graphics cards.

This means that from a computing power perspective alone, High-Flyer may have even obtained the ticket to build a ChatGPT-like model earlier than many large tech companies.

However, large models have strong dependencies on computing power, algorithms, and data. Starting requires $50 million, and each training run costs tens of millions. Companies without billions in capital find it hard to keep up. Despite all these difficulties, High-Flyer remains optimistic. Founder Liang Wenfeng told us: “The key is that we want to do this, and we can do this. Therefore, we are one of the most suitable candidates.”

This enigmatic optimism stems first from High-Flyer’s unique growth path.

Quantitative investment is an import from the United States, which means that almost all leading Chinese quantitative hedge fund founding teams have, to varying degrees, experience at U.S. or European hedge funds. High-Flyer is the sole exception: it was built entirely by a local team, growing through independent exploration.

In 2021, just six years after its establishment, High-Flyer reached a scale of 100 billion RMB and was called one of the “Four Titans of Quantitative Finance.”

Having grown as an outsider, High-Flyer has always seemed like a disruptor. Multiple industry figures told us that High-Flyer “continuously enters this industry with a brand-new approach, in terms of both its R&D system and its products and sales.”

The founder of a leading quantitative hedge fund believed that High-Flyer has “never followed any conventional path,” but rather “pursued the way they wanted to.” Even when it came to unconventional or controversial approaches, they “dared to openly state their views and then act according to their own thinking.”

Regarding High-Flyer’s growth secrets, the company attributes them to “hiring a group of inexperienced but high-potential people, and creating an organizational structure and corporate culture that allows innovation to happen.” They believe this is also the secret that allows their large model startup to compete with large tech companies.

And the more critical secret may lie in High-Flyer’s founder, Liang Wenfeng.

While still studying artificial intelligence at Zhejiang University, Liang was absolutely convinced that “artificial intelligence would definitely change the world.” And in 2008, this was still a conviction few shared.

After graduation, he did not follow his peers to become a programmer at a large tech company. Instead, he retreated to a cheap rental apartment in Chengdu, constantly experiencing setbacks from trying various scenarios, until he eventually entered finance—the most complex of all—and founded High-Flyer.

An interesting detail: in those early years, a similarly eccentric friend who was in Shenzhen’s urban village building “unreliable” aircraft invited him to join. That friend later built a company worth 100 billion USD: DJI (大疆).

Therefore, beyond the topics of money, talent, and computing power that are inevitably involved in building large models, we also had a special conversation with High-Flyer’s founder Liang Wenfeng about what kind of organizational structure enables innovation, and how long human passion can last.

After over a decade of entrepreneurship, this is the first time the rarely-seen “tech nerd” founder has given a public interview.

Coincidentally, on April 11, when High-Flyer announced its decision to build large models, it also quoted advice from French New Wave director François Truffaut to young directors: “Be madly ambitious, and be madly sincere.”

Below is the conversation:

“Do the Most Important, Most Difficult Things”

An Yong Waves (暗涌Waves, a 36kr media brand): Recently, High-Flyer announced its decision to build large models. Why would a quantitative hedge fund do such a thing?

Liang Wenfeng: Our work on large models has nothing directly to do with quantitative trading or finance. We built a completely independent new company called DeepSeek to do this.

Many in High-Flyer’s core team work in artificial intelligence. At that time, we tried many scenarios and eventually entered finance, which was sufficiently complex. Artificial general intelligence is likely one of the next most difficult challenges. So for us, it was a question of how to do it, not why to do it.

An Yong Waves: Will you train a large model yourself, or a large model for a specific vertical industry—such as finance?

Liang Wenfeng: What we want to build is artificial general intelligence, or AGI. Large language models may be the inevitable path to AGI and have already begun to exhibit AGI characteristics. So we will start from here, and there will also be vision and other areas later.

An Yong Waves: Because of the entry of large tech companies, many startups have abandoned the direction of building general-purpose large models.

Liang Wenfeng: We will not prematurely design applications based on the model. We will focus on the large model itself.

An Yong Waves: Many people believe that for startups, it is no longer a good time to enter the field after large tech companies have reached a consensus.

Liang Wenfeng: It currently appears that neither large tech companies nor startups can quickly establish a technological advantage that crushes their competitors. With OpenAI paving the way, and everyone based on public papers and code, both large tech companies and startups will have their large language models ready by next year at the latest.

Both large tech companies and startups have their own opportunities. Existing vertical scenarios are not in the hands of startups, making this phase less friendly to startups. But because these scenarios are ultimately dispersed, fragmented small demands, they are more suitable for agile startup organizations. In the long run, the barrier to large model applications will continue to lower. Startups will have opportunities at any point in the next 20 years.

Our goal is also very clear: we will not do vertical applications, but instead focus on research and exploration.

An Yong Waves: Why do you define your approach as “doing research, doing exploration”?

Liang Wenfeng: It’s curiosity-driven. From a distant perspective, we want to verify some hypotheses. For example, we understand that the essence of human intelligence may be language itself—human thinking may simply be a process of language. You think you are thinking, but it might actually be you weaving language in your mind. This means that human-like artificial intelligence (AGI) could emerge from language large models.

From a closer perspective, GPT-4 still has many unsolved mysteries. While we replicate it, we will also conduct research to unravel those mysteries.

An Yong Waves: But research means paying a higher cost.

Liang Wenfeng: If we only replicated, we could build on public papers or open-source code, needing only to train very few times—or even just fine-tune—which would be low cost. But doing research requires various experiments and comparisons, needing more computing power and higher-quality personnel, so the cost is higher.

An Yong Waves: Where does the research funding come from?

Liang Wenfeng: High-Flyer is one of our funders, with sufficient R&D budget. Additionally, we have a donation budget of several hundred million RMB per year, which has previously gone to charitable organizations. If needed, we can make some adjustments.

An Yong Waves: But building a foundational large model requires at least $200-$300 million just to sit at the table. How do you sustain such continuous investment?

Liang Wenfeng: We are also talking with different investors. From our interactions, many VCs have concerns about doing research. They have exit requirements and want to quickly bring products to market. According to our approach of prioritizing research, it is difficult to obtain financing from VCs. But we have computing power and an engineering team, which is equivalent to having half the chips on the table.

An Yong Waves: What business scenarios and visions have you explored?

Liang Wenfeng: What we are thinking now is that later we can make most of our training results publicly available and shared, which can be combined with commercialization. We hope that more people—even a small app—can use large models at low cost, rather than having the technology controlled by only a few people and companies, forming a monopoly.

An Yong Waves: Some large tech companies will also provide services later. What will be your differentiated part?

Liang Wenfeng: Large tech companies’ models may be bundled with their platforms or ecosystems, while ours will be completely free.

An Yong Waves: Regardless, it is somewhat crazy for a commercial company to pursue an infinitely invested research exploration.

Liang Wenfeng: If you must find a commercial reason, it might be impossible to find one, because it is not cost-effective.

From a business perspective, basic research has a very low return on investment. When OpenAI’s early investors put in money, they certainly did not think about how much return they would get back—they really wanted to do this thing.

What we are relatively certain of now is: since we want to do this thing, and we have the ability to do it, at this point in time, we are one of the most suitable candidates.

Ten-Thousand-GPU Reserves and the Cost They Carry

“An exciting thing perhaps cannot be measured purely by money.”

An Yong Waves: GPUs are the scarce resource in this ChatGPT startup wave. You foresaw this and secured 10,000 cards back in 2021. Why?

Liang Wenfeng: Actually, from the earliest 1 card, to 100 cards in 2015, to 1,000 cards in 2019, and then to 10,000, this process happened gradually. Before a few hundred cards, we hosted them in IDCs. When the scale grew larger, hosting could no longer meet our requirements, so we started building our own server rooms.

Many people assume there is an unknown business logic behind this. But actually, it was mainly curiosity-driven.

An Yong Waves: What kind of curiosity?

Liang Wenfeng: Curiosity about the boundaries of AI capabilities. For many outsiders, this ChatGPT wave was particularly shocking. But for insiders, the shock from AlexNet in 2012 had already ushered in a new era. AlexNet’s error rate was far lower than other models at the time, reviving neural network research that had been dormant for decades. Although specific technical directions kept changing, the combination of model, data, and computing power remained constant. Especially when OpenAI released GPT-3 in 2020, the direction was clear: it required massive computing power. But even in 2021, when we invested in building Firefly No. 2, most people still could not understand.

An Yong Waves: So you started paying attention to computing power reserves from 2012?

Liang Wenfeng: For researchers, the thirst for computing power is endless. After small-scale experiments, one always wants to do larger-scale experiments. After that, we would consciously deploy as much computing power as possible.

An Yong Waves: Many people think that building this computing cluster is for quantitative private fund business, using machine learning for price prediction?

Liang Wenfeng: If we only did quantitative investment, very few cards would be needed to achieve the goal. We did extensive research beyond investment, wanting more to understand what paradigm could completely describe the entire financial market, whether there is a more concise expression, where the boundaries of different paradigms lie, and whether these paradigms have broader applicability.

An Yong Waves: But this process is also just burning cash.

Liang Wenfeng: An exciting thing perhaps cannot be measured purely by money. It is like buying a piano at home: first, you can afford it; second, there is a group of people eager to play music on it.

An Yong Waves: Graphics cards typically depreciate at a rate of 20%.

Liang Wenfeng: We have not calculated it precisely, but it should not be that much. Nvidia graphics cards are hard currency. Even many years old cards, there are still many people using them. When we previously retired old cards and sold them second-hand, they were quite valuable. We did not lose too much.

An Yong Waves: Building a computing cluster, maintenance costs, labor costs, and even electricity bills are all considerable expenses.

Liang Wenfeng: Electricity and maintenance costs are actually very low. These expenses account for only about 1% of the hardware cost per year. Labor costs are not low, but labor costs are also an investment in the future—the company’s greatest asset. The people we select tend to be relatively unpretentious, curious, and come here with the opportunity to do research.

An Yong Waves: In 2021, High-Flyer was among the first companies in Asia-Pacific to obtain A100 graphics cards. Why did you get them earlier than some cloud providers?

Liang Wenfeng: We did early research, testing, and planning for new cards. As for some cloud providers, as far as I know, their previous demands were scattered. It was not until 2022, with autonomous driving and the need to rent machines for training, along with the ability to pay, that some cloud providers went ahead and built the infrastructure. Large tech companies find it hard to purely do research and training; their approach is more driven by business needs.

An Yong Waves: How do you view the competitive landscape of large models?

Liang Wenfeng: Large tech companies certainly have advantages. But if they cannot apply quickly, large tech companies may not necessarily persist either, because they need to see results more.

Top startups also have solid technology, but like the previous wave of AI startups, they all face the challenge of commercialization.

An Yong Waves: Some people feel that a quantitative hedge fund emphasizing its own AI work is just blowing bubbles for other businesses.

Liang Wenfeng: But actually, our quantitative fund has basically stopped raising from external sources.

An Yong Waves: How do you distinguish between AI believers and speculators?

Liang Wenfeng: Believers will be here before, and they will still be here after. They are more likely to buy cards in bulk or sign long-term agreements with cloud providers, rather than renting in the short term.

How to Make Innovation Truly Happen

“Innovation often arises on its own, not through deliberate arrangement, and certainly not through teaching.”

An Yong Waves: How is the recruitment progress for the DeepSeek team?

Liang Wenfeng: The initial team has been assembled. In the early stage, due to insufficient manpower, we will temporarily borrow some people from High-Flyer. When ChatGPT 3.5 went viral at the end of last year, we started recruiting. However, we still need more people to join.

An Yong Waves: Talent for large model startups is also scarce. An investor said that suitable talent may only be in the AI labs of giants like OpenAI and Facebook AI Research. Will you recruit such talent from overseas?

Liang Wenfeng: If you pursue short-term goals, finding people with existing experience is the right approach. But if you look at the long term, experience is not that important. Foundational abilities, creativity, passion, and so on are more important. From this perspective, there are plenty of suitable candidates domestically.

An Yong Waves: Why is experience not that important?

Liang Wenfeng: It is not necessarily people who have done this thing before who can do it. High-Flyer has a principle for hiring: look at ability, not experience. Our core technical positions are mainly filled by fresh graduates and people with one to two years of experience.

An Yong Waves: On innovative businesses, do you think experience is an obstacle?

Liang Wenfeng: When doing something, experienced people will instinctively tell you it should be done this way. But inexperienced people will repeatedly explore and think seriously about how it should be done, then find a solution that fits the current actual situation.

An Yong Waves: High-Flyer, as a complete outsider without any financial background, entered this industry and reached the top within a few years. Is this hiring principle one of the secrets?

Liang Wenfeng: Our core team, including myself, initially had no quantitative experience. This is quite special. It may not be called the secret to success, but it is one of High-Flyer’s cultures. We do not deliberately avoid experienced people, but we focus more on ability.

Take sales as an example. Our two main salespeople are complete newcomers to the industry. One previously did foreign trade in German machinery products; the other used to write code in the back office of a securities firm. When they entered this industry, they had no experience, no resources, and no accumulation.

And now we may be the only large private fund that can rely primarily on direct sales. Doing direct sales means not having to share fees with middlemen. With the same scale and performance, profit margins are higher. Many companies have tried to imitate us, but none have succeeded.

An Yong Waves: Why have many companies tried to imitate you without success?

Liang Wenfeng: Because this alone is not enough to make innovation happen. It needs to match the company’s culture and management.

In fact, in the first year they could not accomplish anything, and only in the second year did they start to achieve some results. But our evaluation standards are not quite the same as typical companies. We have no KPIs and no so-called targets.

An Yong Waves: Then what are your evaluation standards?

Liang Wenfeng: Unlike typical companies that focus on customer order volume, our sales commissions are not pre-calculated based on how much they sell. Instead, we encourage salespeople more to develop their own circles, meet more people, and create greater influence.

Because we believe that a trustworthy, honest salesperson who earns customer trust may not be able to get customers to place orders in the short term, but can make people feel that he is a reliable person.

An Yong Waves: After selecting the right people, what approach do you use to get them up to speed?

Liang Wenfeng: Give them important things to do, and do not interfere. Let them find their own way and express themselves.

Actually, a company’s DNA is very difficult to imitate. For example, hiring inexperienced people: how do you judge their potential? After hiring them, how do you help them grow? These cannot be directly imitated.

An Yong Waves: What do you think are the necessary conditions for building an innovative organization?

Liang Wenfeng: Our summary is that innovation requires as little interference and management as possible, giving everyone the space and opportunity to experiment and make mistakes. Innovation often arises on its own, not through deliberate arrangement, and certainly not through teaching.

An Yong Waves: This is a very unconventional management approach. In this case, how do you ensure that a person is efficient and on the path you want?

Liang Wenfeng: We ensure consistent values when hiring, and then ensure alignment through corporate culture. Of course, we do not have a written corporate culture, because all written things would hinder innovation. More often, it is the manager leading by example. When facing something, how you make decisions becomes a standard.

An Yong Waves: In this wave of large model competition, do you think the more innovation-friendly organizational structure of startups is the breakthrough point to compete with large tech companies?

Liang Wenfeng: If you derive what a startup should do according to textbook methodologies, in the current environment, they would not survive.

But the market is changing. The real driving force is often not some existing rules and conditions, but an ability to adapt and adjust to change.

Many large companies’ organizational structures can no longer respond quickly and act quickly. And they are easily bound by past experience and inertia. Under this new AI wave, a wave of new companies will certainly emerge.

The Real Obsession

An Yong Waves: What excites you most about doing this thing?

Liang Wenfeng: To figure out whether our hypothesis is true. If it is, that would be very exciting.

An Yong Waves: What are the must-check conditions for recruitment this time?

Liang Wenfeng: Passion, solid foundational abilities. Nothing else is that important.

An Yong Waves: Are these people easy to find?

Liang Wenfeng: Their passion usually shows up, because they really want to do this thing. So these people are often looking for you at the same time.

An Yong Waves: Large models may be an endless investment. Will the cost make you hesitate?

Liang Wenfeng: Innovation is expensive and inefficient, sometimes accompanied by waste. Therefore, innovation can only emerge after economic development reaches a certain level. When an economy is very poor, or in industries that are not innovation-driven, cost and efficiency are extremely critical. Looking at OpenAI, it also burned a lot of money before succeeding.

An Yong Waves: Do you feel that you are doing something very crazy?

Liang Wenfeng: I do not know if it is crazy, but there are many things in this world that cannot be explained by logic. Just like many programmers who are crazy contributors to the open-source community. After a very tiring day, they still go contribute code.

An Yong Waves: There is a kind of spiritual reward here.

Liang Wenfeng: It is similar to hiking 50 kilometers. Your entire body is exhausted, but your spirit is deeply satisfied.

An Yong Waves: Do you think curiosity-driven passion can last forever?

Liang Wenfeng: Not everyone can be passionate for a lifetime. But most people, in their younger years, can do something wholeheartedly without any selfish motive.

Translation notes: This interview was originally published in Chinese by 暗涌Waves (a 36kr media brand) on May 24, 2023. The translation aims to preserve the original meaning, tone, and nuance of the conversation while ensuring natural English expression.