The Good and Bad of AI in Animation and VFX
Since founding Mechanism Digital in 1996, I've witnessed many tech changes in the visual effects and animation industry. Some innovations, like 3D animation, have really changed how we tell stories, while others have helped with daily tasks without causing a major shake-up. However, artificial intelligence (AI) and machine learning are the biggest leaps I've seen, changing our industry in ways we couldn't have imagined.
AI's impact has been quick and deep, affecting not just visual effects and animation, but also writing, voiceover, and music production. Technology changes faster each year, and predicting what will happen in the next five to ten years is hard. But AI will definitely be both a challenge and an opportunity for the creative industries. It will drive innovation but also raise important questions about the future of human creativity and jobs.
As a business owner, I've always been eager to try out new tools. This curiosity has led us to beta test early 3D software, compositing plugins, augmented reality apps, virtual reality, interactive real-time game engines, and now AI. This article explores AI's role in visual effects, animation, emerging media, and post-production.
Design and Brainstorming –
One of the main ways our studio uses AI is in the design phase, where it acts like a creative partner. Design is crucial and often takes up 25% of a project's time and budget. In this phase, we ask clients many questions to understand their vision to help us focus a project's direction. AI helps by generating different ideas quickly, letting us explore various directions. It can also inspire unexpected ideas, blending styles creatively. For example, mixing a cowboy theme with 80s new wave elements. If an idea doesn't work, we prompt AI for a new one and quickly refine our approach. However, AI can sometimes suggest ideas that are too expensive for the client. It's important for us to guide the direction and not present options beyond the client's budget. Another drawback is AI's random nature, making it hard to produce consistent styles from shot to shot. We've learned to guide AI using traditional digital tools, like hand-drawn sketches or simple 3D layouts, to maintain control over composition and imagery.
Populating Backgrounds of 3D Scenes -
AI can easily create low-detail textured 3D models, although these models are often best used for background elements due to limitations producing very specific requests or technical issues like baked-in lighting and inflexible UV layouts. One of our favorite uses is GenAI can produce beautifully detailed digital matte paintings, and additional AI tools can add limited 3D depth to add subtle camera pan and zoom movements in compositing programs like Fusion, Nuke, and After Effects.
Rotoscoping and Animation -
Rotoscoping is another area where AI shows promise. AI can do an initial pass, creating a rough mask that artists can use for slap-comps and/or refine later. Eventually, AI will probably eliminate the need for shooting on green screen, reducing costs and time or simply adding flexibility to productions after they have been shot.
Surprisingly, we are now seeing tools to create or influence animation, beginning with simple tasks like animated logo builds. AI applies motion to elements based on shape and composition, which could be useful for non-animators or consumers to create basic animation for a logo. As an animation studio, these tools are not useful to us yet, but as AI evolves, I’m sure it will handle more complex animation.
Editing and Post-Production -
In editing scripted content, AI can build rough assembly edits from script supervisors' notes in minutes and tag shots automatically. For interview content, producers can highlight transcript sections, and Adobe Premiere will edit footage instantly, freeing editors to focus on creative tasks and story coherence. Additional tools will remove all the “ums” and pauses, then clean up background audio and even replace flubbed words.
Educational content and interview content can be automatically broken into sound bites, reformatted for different social media platforms, and uploaded automatically. Using these tools in a pipeline, AI services can monitor related articles/tags on social media channels, generate and tag topical content, and respond to trends within minutes by posting topical content. It can measure performance and adjust strategies for better engagement. Content creation studios, which are not usually involved in media buying and ad placement, can now use new tools to automate the posting and monitoring processes so created content won’t require a handoff to another agency.
Challenges and Opportunities -
AI offers a lot of potential but also brings challenges. Young animation artists must adapt to rapidly changing tools, and senior artists must guide juniors while learning new technologies themselves. Remote work further complicates training, highlighting the need for better teleconferencing tools.
AI has already been replacing some jobs in our industry, like background creation and early design development. This doesn’t mean a computer has taken over their job, but talented artists employing AI can now output better work faster, so the industry needs less artists. This shift is inevitable, and those affected must embrace new technologies to stay current. AI needs content experts to guide its output, ensuring quality and strategy alignment. In the short-term, experienced producers with a trained eye will benefit most, amplifying their design abilities. This will be a hurdle for junior artists entering the industry who haven’t yet developed their “eye”. I believe this imbalance will shift as new generations embrace AI in ways the current generation couldn’t have predicted.
Another downside of AI is its ease of use, which means more people can create content, but this also leads to a flood of mediocre work. It's as if alchemy was discovered and suddenly everyone could make gold, devaluing the precious metal and making it gaudy. I trust our use of AI's "alchemy" will mature and help us create useful materials, making us more productive instead of just showing off.
Embracing AI is essential for staying competitive in media content production. AI speeds up tasks, opens new creative possibilities, and democratizes content production. However, it needs guidance from experienced professionals to maximize its potential. By finding the intersection between personal expertise and AI, individuals can leverage their knowledge to navigate this evolving landscape and harness AI's power for innovative content creation.