vpalos.com // milk, cookies, segfaults…

Loomiere/Stream – A high performance streaming server

by Valeriu Paloş on January 30, 2010

UPDATE: http://vpalos.com/1165/loomiere-2-0-1-beta-finally-out/

The Loomiere (0.2.1) code is now freely available under GPLv3.
Please see this post for an update.

Are you killing psstream?

Well, yes! I am sure that many of you already know about psstream (the PHP streaming extension I made a while back). Well, many things happened since then and I came to realize I could do better; a whole lot better actually. As of now the ‘psstream’ project is officially no longer developed (see below). It will remain on the website for some time to come for archiving purposes but that is it.

But wait, why did you do it?

For some time now I have be looking into improving the streaming mechanism for a large video-sharing project run by my company. PSStream was a first effort, and it did the job but soon ran into problems. None of our servers was able to properly stream more than 150 clients simultaneously and the resources were grossly wasted, hence, Loomiere/Stream.

Care to share some caracteristics?

Sure. Unlike psstream (in which the moov parsing code from code-shop.com was used) Loomiere/Stream is written from scratch in C (and a bit of Bash) and uses a very minimalistic and lightweight approach. A lot of time was invested into optimizing this streamer; here are some delicious features I consider worth mentioning:

  • Secure and configurable
    • Loomiere/Streamer is, in fact, a set of micro-streamers (mp4, flv etc.) that come together into a fast and secure HTTP server using the ipsvd and runit server tools which are very well optimized and stable. This means that I (and you) don’t have to go through the huge pain of making this server secure over time. It also means that the server’s security rules will be very flexible and easy to manage.
  • Optimized to the extreme
    • Each micro-streamer is an extremely small program written in C/dietlibc (the mp4 streamer has at most 21k depending on system) and completely statically linked (no external libraries are linked when executed); this means that these processes can be started very quickly and the CPU is not strained by linking.
    • The micro-streamers have an extremely low memory footprint (mp4 has uses about 120k, never more);
    • Loomiere/Stream uses a “One process per request” approach, which is not usually very recommended on a web-server (due to the fact that it a process is usually heavier than a thread), but in the case of video streaming processes are better (IMO) than thread-pools. This is because each connection transfers a large amount of data and takes a very long time to complete, and would therefore fill-up the thread-pool rather quickly and keep it occupied for the duration of the streaming regardless of server load; here the process start-up time becomes irrelevant, and anyway the Loomiere/Strem micro-streamers startup extremely fast.
    • As a side-note, I think it would be beneficial to test a thread-based approach in the case of smooth-streaming, since in that case the requests would be short enough to make it feasible. Or maybe I could use unlimited threads to stream (i.e. use threads but not a thread pool), that would be worth trying out, I think since it is possible that less context switching will occur. I’ll have to do some tests. :)
  • Saves bandwidth (throttling)
    • The streamer tries to send only enough data to the player so that it will always be up-to-date. This is, of course, configurable via a set of parameters.
  • Versatile
    • Loomiere/Stream is able to serve MP4 and FLV files (flv, f4v, mp4, m4v, mov…) as well as image files (jpg, png, gif, bmp). Images are supported because you may have images to be served along with your movies on the same machine so you won’t have to install a second webserver to access them (however, see the recommendations further down).
    • The starting seek point can be given in bytes or seconds (only MP4 supports seconds yet, but FLV players always seek by byte position anyway).
    • The whole streaming system is very configurable via ‘/etc/loomiere/stream.conf’, and in real-time (you don’t have to restart the server for new settings to become active).
  • Open source
    • Loomiere/Stream is released under the GPLv3 license with an important amendment: one can never use this software for streaming pornographic or sexual content of any kind! Aside from that, you’re clear :).

Did you test this anywhere?

The Loomiere/Stream server entered full production on Friday 29th of January, 2010 at 14:31 on all the content streaming servers of the peteava.ro website. Until now it has been running flawlessly.

A random check (while I’m writing this post) on one of the most strained servers yielded 653 simutaneous clients and a workload of: 8.81, 9.03, 22.47. The old streamer served at most 150 clients with extreme difficulty (the machine was almost frozen). The bandwidth usage dropped from the initial 1300MB/s to about 900 MB/s (approximate values), even though there were many more clients served in parallel.

Everything sounds good but do you think it could be better?

I know it can! Right now the server is in beta state! I can not guarantee anything at this moment (even unofficially) since I am still testing it myself under very heavy load. I will get back with more detailed statistics (and graphs), I promise!

Fantastic, where do I get it from?

ATTENTION: Due to the fact that me and my company are not yet satisfied with some issues regarding the legal terms of Loomiere/Stream, I think it is wise to take it (temporarily) offline until we resolve our concerns!

Giving any support?

I plan to provide commercial support for all Loomiere components (just the streamer for now), so yes! Soon I promise! How soon depends on how many people will use this server and request this specifically.

Until then, the support is basically open and free to anyone who wants to get involved. Practically, if you will have a problem with it and need my help, I’ll be here, I promise!

How do we use it?

The server: I don’t have a proper documentation written yet (that will be coming) but there is a “DETAILS” file in the source archive (above) that explains (in short) what you need to do to get it working. However you should have some knowledge about how ipsvd and runit work before tackling the streamer.

The clients: That’s easy, from any flash video player, you simply use the following url syntax:

http://yourserver.com/path/to/file.mp4 [ ?start=x.xxx [ &end=y.yyy ]]

Is there anything else?

I did not do any benchmarks against the Nginx and Lighttpd modules provided by the people at code-shop.com (they have renewed their modules as well). However, I hope my streamer will perform better considering that it was written with performance in mind, as a standalone server, thus not  being required to work with existing web-servers. Maybe someone could do some tests (maybe I’ll get around to it some day).

Well, for now, give me any feedback, bug reports or feature requests as a comment here.

Remember, streaming high-volume video is a very tricky business, there are many things involved! So if you have a single server that does other things beside the video streaming, I advise you to use your main web server as a front, redirecting streaming requests to the Loomiere/Stream server or simply stream your videos directly on a different port (like 81, or 8080) so that both servers can live on the same machine.

Although Loomiere/Stream is able to stream images, I only recommend you use it in case of large images and not for very small images that are heavily accessed (which is usually the case on the web). So if you can, use another web server for small images (with caching maybe?). I recommend: thttpdnginx or fnord but any other (even Apache) would do just fine (depending on your requirements).

That’s it for now, I gotta sleep. Enjoy! :)

12 thoughts on “Loomiere/Stream – A high performance streaming server

  1. hey great article

    its great that you found a way around to the server
    installations of the secure mods.

     

    I am a web developer and my client requested that the users
    should not be able to download the media files on the site. Now this took a lot
    of time, so I researched a lot on the web for an answer.

     

    Once it was accomplished I decided to write a blog for it.

    http://goldprogramming.blogspot.com/2011/11/secure-streaming-of-media-on-your-site.html

  2. Fhidalgo says:

    Hello, in first time: very thanks for new loomierestream version, i think that i will can to do streaming and control access now, but i have one problem…

    I am trying access to my videos from: myurl:19999/video.mp4 and it show video with size 0kbs, in log file: /var/log/loomiere/stream/current no errors for me, i have setted path to my videos ok (/var/www/videos), but i dont can see it.

    Too, i have tried from flash player access to videos too, and it neither work, you can see example here: http://lospelayospoker.com/demo.html

    Plase any suggest? thanks and regards!

  3. ruthe says:

    +1 Laurie

    We need some explanations please ^^

  4. Laurie says:

    Thanks for the suggestions valeriup. I’ve managed to get the server script running under the xinetd setup. In the browser I can … http://localhost:600/ and get the contents of a video file. Before tidying anything up (script wise) next step is to get the video displayed in a player.

    HISTORY:

    For sometime I’ve been showng the family videos on my site using .flv files, encoded with flv, mp3 audio. xmoov.php to perform streaming and JW Player 4.1. All working well, performance over the WEB OK, all fine and dandy.

    Then came JW player 5.1 with x264 playback. Mmmm, an encode test with x264 for the same bitrate gave quite an improvement. Needless to say x264 in an flv container wouldn’t work properly. Next snag jitter when using ffmpeg to encode, but was OK when using mencoder, the story goes on as you well know …….

    Finally got a good .mp4 file, but then couldn’t use my existing xmoov.php any more (no video displayed). next step try code shops module for apache, and all seemed fine when run locally, but 4 miles down the road over the net a 15 second delay occurred for every seek, back to the drawing board (MP4Box another story), etc, etc ….

    Started on psstream but then I came across loomiere which I liked the sound of, fast, standalone. Before I finalise my setup here (and put a page together for other frustrated soles) I want to be sure JW player 5.1 is going to work, needless to say the couple of attempts I’ve had so far have failed, disappointment is fast becoming my new friend!

    What I need now (to keep away the red herrings) is a proven working example of the syntax required on a web page for JW Player 5.1 using loomier server. Do you know where I can find such an example? Alternatively, maybe I could use the player you have in the demo if JW is known to be a problem for whatever reason. What player is being used on the demo site? Going to switch the screen off for a couple of days now, but I’ll be back.

    Thanks for your time.

    Laurie

  5. valeriup says:

    It would be very nice to see it running on Fedora as well although I don’t know if there are any binary RPM packages for ipsvd/runit under Fedora. You could try compiling and installing the manually, or using CheckInstall to make packages out of the source code (to keep your system clean):

    RUNIT: http://smarden.org/runit/install.html
    IPSVD: http://smarden.org/ipsvd/install.html

    I looked around a bit as well and found the following package for RUNIT under Fedora (may work for you):
    http://rpm.pbone.net/index.php3/stat/4/idpl/12680114/dir/fedora_other/com/runit-1.8.0-1.i386.rpm.html

    For IPSVD nothing came up though :(. Maybe an RPM built for another distribution will still work since IPSVD only consists of a few executable files placed in the global path and some docs (RUNIT is a bit more complex): http://fr2.rpmfind.net/linux/rpm2html/search.php?query=ipsvd&system=&arch=

    In any event, please let me know if you need more help with this. God speed.

  6. Laurie says:

    Some nice work. I’ve started on setting things up. I’m using Fedora and don’t have the like’s of ipsvd-cdb & /usr/bin/sv. I’ll report back on progress. Keep up the good work.

  7. valeriup says:

    Hi, sorry about the delay, the source is now freely available under GPLv3. See this post for an update.

  8. Rv says:

    Hi, hopefully waiting to see … what it is !
    Even a demo … running …
    It’s been 3 months, you’re stuck with legal concerns …

    A bit long for people following you … give us some news about all this stuff

    Thanks

  9. bmigette says:

    hi, i think I’m looking for something like this, so let us know when this annoying legal stuff will be ok, but as all things that goes this way, I have no big hope it’ll be soon :(

  10. CBY says:

    Hey,
    I just faced the difficulties of h264 streaming. I was testing different methods from rtmp (red5) to apache/lighttpd modules (from code-shop) but I wasn’t satisfied with the results I got. Then I found your psstream extension which seemed to be good enough for me, but as you said, many things happened since it was made.
    I read this post and I knew that Loomiere is what I really need. It’s just doing what it needs to, streaming the media. (A complete media server would be just too much for me, I just want to stream flv and h264.)
    I hope the issues with the legal terms will be solved soon, so I could give it a try.
    Nice job!

  11. valeriup says:

    @Arjen: Thanks a bunch an I think that’s right: It’s impossible for now to make a decent comparison between the two. I will also post some charts showing traffic and system indicators from our streaming servers. Very soon I promise!

    From what I can tell right now (from your charts and what my cluster shows me), Loomiere/Stream yields more throughput and more simultaneous connections but at the same time the CPU has higher iowait times than mod_h264_streaming. I also believe that there is currently more context switching being done in Loomiere/Stream due to the one-process-per-request architecture. I am certain that a lot could be done to balance and fine-tune this streamer.

    I’ll try to get back with charts as soon as I will have some time for it.

    Good job, and also Godspeed on your projects,

    Valeriu

  12. arjen says:

    Hi,

    Interesting read!

    Please find a benchmark with some figures/pictures of the mod_h264_streaming module on the following page: http://h264.code-shop.com/trac/wiki/Mod-H264-Streaming-Performance-Version2

    It’s difficult to compare the figures from the two modules as there are a lot of variables (system, webserver, average duration of the videos, filesize of the videos, etc..

    When I have time I’ll install your module and try to do a comparison.

    Good luck with your projects,

    Arjen

Don't keep it to yourself!...