Partnered With:

a16z: An overview of the market ecology of generative AI and gaming

a16z crypto

Read this article in 25 Minutes

This article will introduce the a16z investment layout of the circuit.

The Generative AI Revolution in Games

Original article by James Gwertzman and Jack Soslow

Notes on the study of Alpha Rabbit

A16Z 最近写了一篇很有意思的文章，谈到他们认为的生成式 AI 和游戏结合在一起的机会在哪，笔者翻译后对部分内容进行了注解。文章第一部分已经发出，见：《 A16Z | 生成式 AI 在游戏领域的机会（市场假设+预测）》，本篇为第二部分，包括 A16Z 对游戏 + 生成式 AI 领域的市场生态的判断（请注意：这些大部分都是 A16Z 的 Portofolio，请大家本着客观理性的态度阅读）。

Overview of market ecology

The chart below shows the overall market ecology, depicting the startups A16Z found in each category. In each of these specific projects, we looked at the impact of generative artificial Intelligence (AIGC) on games. This article will look at the most distinctive companies and opportunities in each category.

Generate 2D Images with text

Generating 2D images from text prompts is one of the most widely used areas of generated AI. Tools such as Midjourney, Stable Diffusion, and Dall-E 2 directly transform text descriptions into high-quality two-dimensional images, and they are used at various stages throughout the life cycle of game development and production.

(Note: Midjourney is also an easy to use AI image generator. It's easier to get started. It generates images very quickly, producing 4 images in 1 minute.)

Concept Art

(Note: Concept art can also be called preliminary design, and is often used as a professional design concept in the film, television or game industries. In general, it means to set the tone for the visual effects of a product. It is one of the first core tasks of a game or film. Through new design ideas and directions (including modeling, spirit, concept, etc.), innovation and even subversion of the past way, to create a new modeling role or concept.

What's the difference with illustration? The relationship between an illustrator and a game is more about drawing posters, wrapping covers, etc. And one of the 2D Art work that is really closely related to the work of film, television and game development is what we call Concept Art. What's the difference with comics? Concept Art is very different from manga. Manga (manga) is an independent system in Japan, which serves more for the visualization of popular light novels. While concept art serves games, animation is not restricted in style and form, and it is extremely important to create some advanced or complete sets of Settings. What's the difference from the original painting? Concept Art includes almost all roles and scenes. Different from the original artists, concept designers must take the initiative to produce some more interesting designs.

Different from original artists, Concept Art plays an artistic decision-making position next to the master planner in games and animation projects, which determines the style and audience of the project. Therefore, a painter who aims at conceptual artists and is confined by one painting style is incompetent.

Reference: public number a Sajun: the history of the most complete! Share by Yufei Zhang, ConceptArt Major of Saide University --2020-02-04 18:00

Generative AI tools are useful in helping a role like a game designer explore game concepts and generate ideas. This is also a key part of the production process. For example, a game studio is using the above tools to radically speed up their concept art development process because they create an image in just one day, whereas previously the process would have taken up to three weeks, but how?

First, game designers use Midjourney to explore different inspirations and generate concept images as they see fit. After that, the images are given to a professional concept artist, who can put them together and then create a coherent image of a related topic, which is then fed into Stable Diffusion to form a series of image changes.

We discuss these different styles, decide on one, edit it manually with a brush, and repeat the process until we are satisfied with the result. At this stage, the image is then uploaded to a Stable Diffusion to create the final artwork.

2D Production Art

Other game studios are experimenting with similar AI tools to create in-game art. For example, here is a tutorial from Albert Bozesan on how to use Stable Diffusion to create 2D assets in a game.

Source: https://www.youtube.com/watch?v=blXnuyVgA_Y

3D Artwork

The 3D stereoscopic module is an important building block for all modern games today and the upcoming meta-universe. Virtual worlds and game levels are essentially a collection of 3D assets that fill the game environment with different parameters modified through different combinations and placement methods. Creating a 3D element is more complex than creating a 2D floor plan and involves several steps, including the need to make a 3D model, add textures and effects. For an animated character, it also involves creating an internal "outline" and then creating an animation on top of that outline.

We found a variety of startups looking for opportunities in all stages of the 3D asset creation process, from model building to character animation to level building. However, this part of the business and innovation is still being explored.

3D assets (3D assets)

Startups trying to move toward creating 3D models include Kaedim, Mirage and Hypothetic. Big companies are also looking at the issue, including Nvidia's Get3D and Autodesk's ClipForge. Kaedim and Get3d focus on the conversion of images to 3D models; ClipForge and Mirage focus on text-to-3D conversion, while Hypothetic is interested in both text-to-3D search and image-to-3D.

Kaedim: Based in London, it generates 3D models from 2D images.

Mirage: https://www.mirageml.com/

3D Textures

3D models can be more realistic if they can be used on grid-based textures or materials in a game. For example, using different types of mossy wind fossils on a medieval castle model can completely change the appearance of a scene. Textures, which contain metadata about the light's reaction to the material (i.e., roughness, glossiness, etc.), can be easily generated by artists based on text or image cues, and are valuable for improving the speed of iteration in the creative process. Companies like BariumAI, Ponzu and ArmorLab are working in this area.

BariumAI:https://barium.ai/

Ponzu:https://www.ponzu.gg/

animation

The production of good animation is one of the most time-consuming, expensive and skillful parts of the game creation process. One way to reduce costs and create more realistic animation is to use motion capture, which is to dress actors or dancers in motion capture suits and record their movements with specialized equipment.

We found that current generative AI can capture animation directly from video. This is much more efficient because there is no need for expensive motion capture equipment, and it means we can capture animation from existing video.

Another exciting aspect of the AI model is that it can be used to filter existing animations and add new special effects, such as making an animated character look drunk, or old, or happy. Companies in this space include Kinetix, DeepMotion, RADiCAL, Move Ai and Plask.

Kinetix

DeepMotion

RADiCAL

Level design & Level Design & Amp; world building)

One of the most time-consuming aspects of game creation is building the game world, and generative AI can be used for this task. Games like Minecraft, No Man's Sky, and Diablo are known for procedurally generated levels, where the levels are generated randomly, different each time, but follow the rules laid down by the level designer. One of The big selling points of The new Unreal 5 game engine is its collection of programmatic tools for open world design, such as leaf placement.

Companies like Promethean, MLXAR, or Meta's Builder Bot are seeing opportunities in generative AI. There's been a lot of academic research on this for a while, from Minecraft generation techniques to Doom level design.

Why do generative AI tools have the potential for game level design? Because AI has the ability to create different styles of levels and game worlds. Imagine tools that quickly generate a game world in the heady New York of 1920, or the design of the mysterious dystopian Blade Runner, Or the fantasy world of Tolkienism (similar to the design and landscape of the Ring) (vs dystopian Bladder-runner -esque future, vs. Tolkien-esque fantasy world.).

The following concepts are the different styles of levels in the game generated by Midjourney using hints:

audio

Sound and music are an important part of the experience. Companies are already using generative AI to generate audio to complement their graphics work.

Sound effect

Sound effects are another attractive area for AI. Academic papers have explored the idea of using AI to generate "foley" in movies (footsteps, for example), but there are few commercial products that can be used directly in games.

I think it's only a matter of time, because the interactive nature of games makes them an obvious application for generative AI, both to create static sound effects as part of production (" laser gun sounds in games, etc. ") and to create real-time interactive sound effects at runtime.

Imagine generating footsteps for the player character (e.g. CS and chicken footsteps...) ? Most traditional games solve this problem with a small number of pre-recorded footsteps: for example, walking on grass, walking on gravel, running on grass, running on gravel, etc. These sounds are cumbersome to publish and manage, and sound repetitive and unreal when run.

A better approach is to use generative AI simulation sound effects in real time to produce appropriate and more realistic sound effects through the game's parameters, such as the ground, character, weight, gait, footwear and other different media, to represent different sound effects.

Music (game soundtrack)

Music is important for games because it helps set the emotional tone for the theme of the story, just like in a movie or on television. But because games last longer, sometimes for hundreds or even thousands of hours, unchanging music can quickly become repetitive or annoying. Moreover, due to the interactive nature of the game, it is difficult for the soundtrack to fully match the random scenes and actions occurring on the screen.

Adaptive music has been a topic of interest in game soundtracks for more than two decades, dating all the way back to Microsoft's "DirectMusic" system for creating interactive music. DirectMusic hasn't been widely adopted, though, mainly because of the difficulty of composing in the format, and only a few games, such as Monolith's And Then There Were None, have created a truly interactive soundtrack (Monolith "s No One Lives Forever,).

Right now, there are a number of startups trying to create AI-generated music, such as Soundful, Musico, Harmonai, Infinite Album, and Aiva, although many of the current tools, like Open AI's Jukebox, are highly computation-intensive, It will not run in real time yet, but it will become possible once the initial model is successfully built.

Dialogue & Speech and Dialog

Many companies are trying to create realistic sounds for characters in games, and of course, given the long history of speech synthesis in computers, this is not uncommon: Sonantic, Coqui, Replica Studios, Resemble.ai, Readspeaker.ai, and many more. There are several advantages to using generative AI for speech, and of course, the race is competitive.

Instant dialog generation. More often than not, the game's voice is pre-recorded by voice actors, but these are limited to rigid speeches. With generative AI dialogue, characters can say anything, which means they can react adequately to the player's actions.

Role play. Many players want to play as virtual characters that bear little resemblance to their real-world identities. However, this illusion is shattered as soon as the player speaks in his own voice, and the use of a generated voice that matches the player's avatar can maintain this illusion.

Control sound effects. When generating speech with AI, we can control the nuances of the voice, such as its intonation, twist, emotional resonance, phoneme length, accent, and so on. Localization (convenient translation and foreign promotion). Conversations can be translated into any language and speak in the same voice, and companies like Deepdub specialize in this niche.

NPC& Player character

Many startups are investigating the use of generative AI to create interactive characters, and in addition to the market opportunity for NPCS in games, there is plenty of room for growth for virtual assistants or receptionists. The effort dates back to the early days of AI research.

Many companies are building general-purpose chatbots, many of which are powered by language models similar to GPT-3. A handful of companies that are specifically trying to build chatbots for entertainment purposes, such as Replika and Anima, are trying to build virtual chaperones. The era of virtual girlfriends, as seen in Spike Jonze's sci-fi romance "Her," starring Joaquin Phoenix, Scarlett Johansson and others, may soon be upon us.

You can now see the next iteration of these chatbot platforms, such as Charisma.ai, Convai.com or Inworld.ai, which can render 3D characters for power as well as emotional display, with tools that allow creators to set goals for these characters, It can have a narrative status as an integral part of the game or in advancing the plot, rather than a mere ornament.

Integrated platform

The most successful generative AI tools, like Runwayml.com, can bring a wide range of creator tools into one. However, there is no such company in the gaming space yet, and A16Z would like to invest in a generative AI gaming solution with the following characteristics:

A full suite of generative AI tools covering the entire production process (code, asset generation, textures, audio, etc.) can be used in conjunction with popular game engines such as Unreal and Unity using a generic scenario suite of tools designed to fit into typical game production

summarize

It's an incredible time to be a game creator. One day, we can imagine games that are personalized and customized for players, based solely on their needs.

This kind of scenario has been around in science fiction for a long time -- like the "artificial intelligence puzzles" in Ender's Game, or the holodeck in Star Trek -- and it's all coming to life in the future.