Giters

jbilcke-hf / clapper

Clapper.app, a video synthesizer and sequencer designed for the age of AI cinema

Home Page:https://clapper.app

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add a new workflow category for lip syncing (VIDEO_LIPSYNC)

jbilcke-hf opened this issue · comments

commented

Context

Since we have a decent way of generating a video baseline (eg. using Live Portrait directly from Clapper),

we need to take those videos and lip-sync them to the dialogue segments

Solution

  • Add a new workflow category for lip syncing (VIDEO_LIPSYNC)
  • Create at least one example workflow

I propose to run the lipsync workflow in the segment resolver (eg. after generating a video or applying a video face swap)

For instance, this can be done after this block:

https://github.com/jbilcke-hf/clapper/blob/main/packages/app/src/app/api/resolve/route.ts#L164-L203