跳到正文
DEV Community· Maham·· 11 天前AI 評分53

PolyCast 作者為何以 Three.js 和 headless Chrome 逐格渲染影片

I render videos frame by frame with Three.js in headless Chrome. Here is why

AI 導讀

Maham 發布免費、MIT 授權的開源工具 PolyCast,使用 JSON 對白腳本生成低多邊形 3D podcast 影片,並將生成語音、口型、鏡頭切換和字幕按字詞時間同步。

正文

Every short video I made cost me an hour in an editor for less than a minute of output. The editing was never the creative part. It was dragging the same cuts to the same words, again and again.

So I asked a different question: what if the video is just data? A script, a list of who says what, and the timings of the words. Everything else, the camera, the captions, the characters moving their mouths, is a rendering problem. Rendering problems I can solve with code.

That became PolyCast, a small open-source tool I just released. Write a script as JSON, and it renders a low-poly 3D podcast video: two characters in a studio, generated voices, lip-sync, and camera cuts and captions cued to the exact word.

How the pipeline works

It runs in this order:

  1. The script. A JSON file with the dialogue lines, who says each line, and camera cues. That file is the whole video, nothing else is edited by hand.
  2. The voices. Each line is turned into audio with Cartesia. If you do not have a Cartesia key, it falls back to your computer's free built-in voice, so the whole thing can run at zero cost.
  3. The timings. Once the audio exists, the tool knows when every word is spoken. Lip-sync is driven by the real audio, and the camera cuts, the screens in the studio and the captions are all cued to those same timings.
  4. The render. The scene is Three.js running in headless Chrome. Instead of recording the screen in real time, it steps the scene forward frame by frame and captures each frame, then assembles the MP4.

Why frame by frame

Screen recording drops frames and drifts. If the machine stutters, your video stutters. Capturing frame by frame is slower, but the output is deterministic: the same script always produces the same video, frame for frame. For a tool that other people will run on machines I cannot see, that mattered more to me than render speed.

A small voice problem I did not expect

The demo is about PolyCast itself, so the characters say "Claude Code" a lot. Cartesia kept pronouncing Claude closer to "cloud". I ran a control test and found the phonetic spelling "Clawd" comes out right every time, so the voice script says Clawd while the captions still read Claude Code. Small thing, but if you generate voices for your own tools, test your product names first. The voice will get them wrong in ways you only hear at the end.

It also works as a Claude Code skill

This is the part I end up using daily. PolyCast can be added to Claude Code as a skill. I describe the video I want in plain words, and it writes the script, generates the voices, directs the scene and hands me the finished MP4. The 44 second demo on the repo was made exactly like that. I never opened an editor.

Try it

PolyCast is free and MIT licensed: https://github.com/mahamsultana/polycast

The demo video, a frame grid and the full guide are in the repo. If you try it, I would like to know one thing: is JSON the right interface for a video script, or would you rather write it in Markdown? That is the decision I am least sure about.

來源:DEV Community · dev.to