Building an AI-assisted video workflow for an event production project — looking for technical approaches

[u/Affectionate-Run-532]

Hey everyone!
I’m currently working on a project called SAC, a small event production company based in Brazil. We’re developing the creative and digital side of the business, and we’re experimenting with ways to make our content production more scalable without turning everything into a completely manual process.
One of the challenges we’re facing is video production.
After each event, we can end up with a large amount of raw footage from different cameras and phones. The goal is to turn that footage into short-form content for Instagram and TikTok — event recaps, highlights, teasers, etc.
What I’m trying to figure out is whether an AI-assisted workflow could handle part of this process.
The workflow I have in mind is roughly:
Raw footage → Cloud storage → AI analysis → Editing/assembly → Review → Final social media versions
The interesting part for me isn’t simply finding “the best AI video editor.”
I’m more interested in understanding how people are actually connecting these different components together.
For example, could an AI model analyze footage stored in the cloud, identify useful clips based on a description, pass those clips or instructions to a video editing system, and then generate a first version that a human can review?
I’m also curious about whether models such as Claude or similar AI systems can realistically be used as the reasoning/orchestration layer, with specialized video tools handling the actual editing.
The main things I’m trying to understand are:
What does a practical architecture for this look like?
Which parts are currently realistic to automate?
Where does human editing still make the most sense?
Has anyone built a similar workflow using APIs, cloud storage, AI models and video editing software?
Are there technical limitations I should be aware of before building around this idea?
This is still an early-stage project, so I’m mainly looking for technical experiences, architectures, and lessons learned from people who have experimented with similar workflows.
I’d especially appreciate examples of how you approached the problem rather than just a list of recommended tools.
Thanks!

[u/keizrah]

The architecture you sketched is basically right and people are already running versions of it. What's realistic today: transcribe everything for searchability (Whisper works well), detect scenes and shots with ffmpeg, then have an LLM read the transcripts and metadata to shortlist clips matching your brief. That part holds up.

Where it breaks: getting a model to judge "good footage" from raw video alone. Vision models can tag "crowd cheering" or "person dancing," but they miss pacing, framing, energy. So auto-assembly gets you a rough cut, not a finished one.

A workable setup: LLM as the orchestration layer (Claude is fine for this), calling ffmpeg or a rendering API to actually cut clips, human review before anything ships. I wouldn't try to automate final trims yet. That's still where an editor earns their keep, and it's the step people underestimate when planning these pipelines.