NavinF
9 months ago
> need a shared filesystem for this to work
Oh oof. I thought removing that requirement would be the whole point of something named "FFmpeg-over-IP". Shared filesystem usually involves full trust between machines, bad network error handling, and setting things up by hand (different config on every distro)
steelbrain
9 months ago
I hear you. If your usecase doesn't require live streaming of converted file, a sibling comment may fit the usecase: https://news.ycombinator.com/item?id=41745593
qwertox
9 months ago
One could write a small Python server in one day which receives a chunked POST request and transcodes the video on the fly.
The same server could also offer a download link for the transcoded video, and also receive URL parameters for the transcoding options. Or the transcoded video itself is returned after the subprocess finishes.
Something along the lines of:
Server
import asyncio
import subprocess
from aiohttp import web
async def transcode_video(request):
cmd = [
"ffmpeg",
"-i", "pipe:0",
"-f", "mp4",
"-preset", "fast",
"output_file.mp4"
]
process = await asyncio.create_subprocess_exec(
*cmd,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
async for chunk in request.content.iter_any():
process.stdin.write(chunk)
await process.stdin.drain()
await process.stdin.close()
await process.wait()
return web.Response(text="Video transcoded successfully!")
app = web.Application()
app.router.add_post('/upload', transcode_video)
if __name__ == '__main__':
web.run_app(app, port=8080)
curl -X POST http://<server_ip>:8080/upload \
--header "Content-Type: application/octet-stream" \
--data-binary @video.mp4
steelbrain
9 months ago
This and the other solutions in this thread wont work with plex or the media servers. The way it works with the media servers is that there’s a variable number of outputs, and you just cant emulate that on the client side without a virtual filesystem or by some creative hacks that use multiple file descriptors (even then how do you handle dynamic?).
As for why the variable number of outputs exist in the first place, it’s because when you’re a media server, you dont want to wait until the full file is converted before serving it to the user. You want to start playing immediately so the output is chunked into multiple files
NavinF
9 months ago
Ah unfortunately my use case is similar to yours: Use Windows desktop to transcode files stored on a Linux NAS. My files are ~100GB so encoding multiple files in parallel would waste a lot of space and unnecessarily burn write cycles
steelbrain
9 months ago
FWIW, you can run an smb server from within a docker container (on the linux side). I forget which one I used but it makes the setup painless and you can configure different auth strategies as well. Network errors (little bit of packet loss) are generally handled by the underlying OS, and in case of windows, it can use multiple network paths simultaneously to give you the aggregate bandwidth.
linuxdude314
9 months ago
You typically don’t even have to go to that length. This is usually supported out of the box.
No idea what problem this is trying to solve. Just seems like the user wasn’t familiar enough with how to use a NAS.
NavinF
9 months ago
> out of the box
If you're talking about how today's open source NAS software has a button for enabling NFS/SMB on a directory:
1. I built my NAS long before software like that was common. Some of my custom stuff (Eg tiered storage, storing the first few seconds of every video on flash, etc) would be a pain to migrate.
2. Some of my Windows machines are untrusted. Unlike the NAS, they have internet access. I can't give them read access to the entire NAS, but I still want to use their GPUs and CPUs to run ffmpeg on arbitrary files on the NAS.
3. I could spend a day writing more code to move files in and out of a shared directory every time I need to run ffmpeg on them. But I was hoping "FFmpeg-over-IP" would let me run a remote ffmpeg on a local file. Like calling an RPC
linuxdude314
9 months ago
Did you build your NAS in the 90’s or early 2000’s?
This has been available in Synology and QNAP devices for that long…
I used to own the tape library that is the Disney Vault. A common pattern for transcoding is to have a watched folder. Drop files you need transcoded in, get files you want in a diff directory when finished.
NavinF
9 months ago
https://news.ycombinator.com/item?id=41748751
Neither the OP nor I are talking about offline transcoding.
oefrha
9 months ago
More like you’re not familiar enough with the video encoding performance of a typical NAS that is not in the thousand dollar range. Or what’s a NAS — it can be anything really.
linuxdude314
9 months ago
Why would you transcode on the NAS? Not what I’m suggesting at all; perf on most NAS is awful for that…
There’s a pretty well understood concept of what a NAS is; this isn’t a complicated philosophical problem.
A very common workflow in motion picture production is to use NAS for storage on a fast network, mount the SMB share, have a script/tool/app that monitors the ingest directory and writes to an output dir.
FWIW the key differentiator between a NAS and other types of network storage is the protocols they use.
If files is the main primitive, it’s a NAS; it’s its blocks it’s considered a SAN.
Sometimes SANs have NAS “heads” for clients that want file access or a block level storage device.
otterley
9 months ago
My five-year-old Synology DS418play has support for hardware realtime transcoding via its integrated low-power GPU. I think I paid about $600 for it. You can get a new DS423+ for about the same price.
NavinF
9 months ago
That only works if your output codec is something really old like H264
otterley
9 months ago
What's the point of transcoding to HEVC? If your playback device already supports it, and the file is already HEVC, then there's no need to transcode. And there's no benefit I'm aware of in transcoding from an older format like H.264 to HEVC. AFAIK, every device that supports HEVC media also supports H.264 as well.
It would be helpful to understand your specific use cases in more detail, particularly for batch transcoding (my understanding is that this solution is not about stream transcoding).
NavinF
9 months ago
Aight have you tried transcoding a normal file (today that means 4K resolution) to H264? It usually won't play because the standard limits resolution to 2048 pixels.
user
9 months ago
yarg
9 months ago
Couldn't you create and share a virtual file system with FUSE?
NavinF
9 months ago
Mounting anything requires root even if you use FUSE.
There are ways to intercept writes without root and send them to another machine. Eg you could use LD_PRELOAD. But that's exactly the kind of pain in the ass that I was hoping a project named FFmpeg-over-IP would automate.
duped
9 months ago
mount doesn't require root, but even without getting into mount namespaces, this is why fusermount3 exists
yjftsjthsd-h
9 months ago
Like SSHFS, or something custom? And how much does that help with any of those concerns?
yarg
9 months ago
The primary concern is trust/security.
User space limits potential security impacts, and a restricted VFS could be used to prevent clients from accessing anything that they shouldn't.
(Although I'm not even pretending to know whether or not this is a remotely good idea - my guess is that it isn't, but I'd like to know just how bad an idea it actually is.)