Open source image, audio, video hosting - MediaCrush

SirCmpwn · 2014-02-04 01:22:04

Hey guys! I've been building MediaCrush over the past few months, with my friend (and fellow Arch user) Jose Diez, and several other open-source contributors. It hosts video, audio, images, vectors - over 500 different file formats are supported. You can upload almost anything (like a mp3, flac, mp4, webm, avi, bmp, svg, cr3...) and it'll be converted (losslessly where possible) to browser-friendly formats. You get a link to share, like this: https://mediacru.sh/Fx73lJhU0_Id

The site is completely open source. You can run your own clone of MediaCrush locally with the help of the detailed instructions in the README. Pull requests are welcome, too, so you can help make it better if you please. It's written in Python and it's built on the shoulders of giants like ImageMagick and ffmpeg. Our whole shop is Arch - our dev machines are Arch, and all of the production servers (web servers, processing servers, and storage servers) are running Arch.

✓ Share images, audio, video, vectors
✓ Super fast hosting
✓ Hotlinking/embedding
✓ Create albums of media
✓ Keep track of files you've uploaded (optional)
✓ Privacy-first data policies (link)
✓ Excellent public API (link)
✓ Entirely open source
✓ HTTPS only
✓ Ad-free

You can upload files right away at https://mediacru.sh, or set up your own instance to host files on. Feel free to hop into the IRC channel for a chat, too: #mediacrush on irc.freenode.net. There are a couple of other cool things that you might like. For example, MediaCrush-cli lets you upload files from the comfort of your favorite shell.

$ mediacrush --album *.jpg # example usage

You can find it in the AUR as mediacrush-cli. There are also browser extensions that make it easy to rehost images around the web: Firefox, Chromium (both are open source, of course 1, 2). I also have a little script that uploads screenshots via scrot straight to MediaCrush.

Here are some example URLs to play with: Video, or audio, or a GIF (we convert those to video, makes them load faster), or how about an album?

Let me know what you think! I hope you like it.

Last edited by SirCmpwn (2014-02-04 01:26:43)

likytau · 2014-02-04 14:14:37

As usual when encountering a new application, I set out to see if I could break it:

https://mediacru.sh/1hBjPzWZGy6m

Contains (many) extra colors over the original.

I can kind of understand this practice :

a GIF (we convert those to video, makes them load faster)

practice, but of course it totally breaks on pixel art.

Here's a larger example (this image is not by me, whereas the earlier one was):

https://mediacru.sh/R41gV6LZwvAd

Artefacts can be observed using a program like xzoom.

Now that that round of black-hatting is over, I will say that otherwise, it seems to work quite well. I would like to generally know more about what exact transformations it does.

* It looks like it runs optipng with default params on input PNGs. PNG in, PNG out, good.

* I haven't got a movie clip to hand to test it with
* Fails on .ora files, but they show up in the file selector. I don't know enough to say whether this is a bug in mediacrush or a part of the browser beyond its control.
* claims to succeed on .xcf files, but the result (https://mediacru.sh/UBACpuKjRXyl) is blank. With the aid of xcf2png, I can show that the result should look like this:

http://i.imgur.com/Iy72Q7V.png

(EDIT: oops, big!)
* svg ✓
* gpl ✗
* ase ✗

It's a bit late, so I'll do more testing another day (high bitdepth png? webp?). I can probably code gpl support if you want it, since I'm familiar with both gpl and Python.

-- mod edit: read the Forum Etiquette and only post thumbnails http://wiki.archlinux.org/index.php/For … s_and_Code [jwr] --

Last edited by likytau (2014-02-04 14:15:20)

drcouzelis · 2014-02-04 14:33:12

Thank you for the contribution.

What was the purpose of creating this website and software? Hosting media takes a lot of processing power and storage space. Is there a limit to how much you will host? Is this website "permanent" or just a demo of your software?" Is this website for profit? Do you have a revenue model? What is the policy on licenses for the media?

(I apologize if this is explained on the site. If it is, I'll read more about it tonight when I get a chance.)

SirCmpwn · 2014-02-04 19:07:19

likytau wrote:

a GIF (we convert those to video, makes them load faster)
practice, but of course it totally breaks on pixel art.

You're absolutely right. The reason we decided to convert GIF to video is because the majority of GIFs are sourced from video, and GIF is a poor choice for that kind of media. However, GIF is well suited to pixel art. The original GIF doesn't go away when you upload - if you want to host GIFs on MediaCrush like that, I suggest hotlinking straight to it: https://mediacru.sh/1hBjPzWZGy6m.gif. That being said, we strive to make the conversion as close as possible to the original, and I feel that in both cases, the final result is close enough to give the general idea, and then you get the original GIF when you click "Download", so it's suitable for sharing.

likytau wrote:

I would like to generally know more about what exact transformations it does.

Files are run through detect, which determines what kind of file they are, collects some metadata, and selects a processor. Processing is distributed, which is accomplished through Celery (with Redis as a broker).

likytau wrote:

Fails on .ora files, but they show up in the file selector. I don't know enough to say whether this is a bug in mediacrush or a part of the browser beyond its control

Part of the browser.

Regarding the other formats you mentioned, could you offer some test files, and a general idea of what they are and how we might go about detecting them and supporting them? You're welcome to submit pull requests; you should join the IRC channel to participate in dev chatter. Thanks for the feedback!

drcouzelis wrote:

What was the purpose of creating this website and software? Hosting media takes a lot of processing power and storage space. Is there a limit to how much you will host? Is this website "permanent" or just a demo of your software?" Is this website for profit? Do you have a revenue model? What is the policy on licenses for the media?

I created it because I wanted a generic media hosting site that didn't suck. We have plans in place to expand our hosting capacity infinitely (obviously at a higher cost), but we're well within our operating limits right now. The website is permanent, we will host whatever you upload until the end of time. The limitations are that no single file can be larger than 50M (at least officially, the actual limit is closer to 100M), and you're rate limited to a certain amount of uploaded files per hour. Regarding revenue, we're working on some options that might drive revenue, but we're not going to do anything that makes it harder on free users. We're thinking about business plans based on our HTML5 video pipeline. Relatedly, we publish transprency reports each month, including financial reporting, on our blog.

Last edited by SirCmpwn (2014-02-04 19:10:37)

DrZaius · 2014-02-04 19:15:47

What does ffmpeg-full from AUR offer that you require that isn't available with ffmpeg from Extra? Not one person in history has ever needed ffmpeg with almost every external library enabled.

Last edited by DrZaius (2014-02-04 19:16:30)

SirCmpwn · 2014-02-04 19:21:29

DrZaius wrote:

What does ffmpeg-full from AUR offer that you require that isn't available with ffmpeg from Extra? Not one person in history has ever needed ffmpeg with every external library enabled.

Well, we want to support every file format that we can, so we enable everything. But to be more specific, we want libx264 (non-distributable), libfdk_aac (non-distributable), and a few others. We also edit the ffmpeg-full PKGBUILD before we actually install it on our production systems, and remove a few things like x11grab (it's been a while since I set up a processing system on production, forgive me if I don't remember all the details).

drcouzelis · 2014-02-04 19:33:22

Thank you for answering my questions. Good luck with your site. I look forward to using MediaCrush for the next "Arch Linux screenshots" thread.

DrZaius · 2014-02-04 22:17:45

SirCmpwn wrote:

But to be more specific, we want libx264 (non-distributable), libfdk_aac (non-distributable), and a few others.

x264 is not non-distributable, and libfdk_aac only needs --enable-nonfree if --enable-gpl is included.

SirCmpwn wrote:

We also edit the ffmpeg-full PKGBUILD before we actually install it on our production systems, and remove a few things like x11grab

If you didn't remove it yet, you can also avoid the sdl dependency since you won't need ffplay. I'd just be wary of using ffmpeg-full as the PKGBUILD base for a production machine.

Have you considered jpegtran too?

jpegtran -optimize -perfect input.jpg > output.jpg

Your ffmpeg commands seem good. For the mp4 output you might want to add "-movflags +faststart" so the video can begin playback before it is complete downloaded by the viewer (assuming the users will host the videos and they will be watched via progressive download).

For .ogv output you could specify the stream(s) that "-q" should be applied to since without any stream specifiers it could be considered ambiguous (-q:a and -q:v).

I saw no instance of libfdk_aac anywhere.

SirCmpwn · 2014-02-04 22:21:00

Thanks for the tips. We'll probably prepare our own PKGBUILD next time we set up a production server. Regarding jpegtran, I'll have a look. I've been looking for good jpg optimization software.

I don't think we really need to allow playback before files finish encoding.

As for -q, you're right. I'll modify it next time I have the code up.

Finally, regarding libfdk_aac, if it's installed, ffmpeg will use it by default when encoding aac. We might want to specify it explicitly for cleanliness, I suppose.

DrZaius · 2014-02-04 22:38:09

SirCmpwn wrote:

Regarding jpegtran, I'll have a look. I've been looking for good jpg optimization software.

jpegtran is great, but it may not give a substantial size decrease in every input. I forgot to add "-copy none" which may be useful. You can even crop losslessly with jpegtran.

SirCmpwn wrote:

I don't think we really need to allow playback before files finish encoding.

Maybe not for you, but for your users if they want to download their converted file and then use it in a fashion that would benefit from having the moov atom at the beginning of the file.

SirCmpwn wrote:

Finally, regarding libfdk_aac, if it's installed, ffmpeg will use it by default when encoding aac. We might want to specify it explicitly for cleanliness, I suppose.

Yes, you should be explicit with all encoders so the defaults don't give you a surprise.

-crf 18 might be considered overkill. The default (-crf 23) would probably be ~30-40% smaller in size (depending on the input) and most users probably won't notice a difference. A faster preset might not hurt either. Users are impatient.

Last edited by DrZaius (2014-02-04 22:40:44)

SirCmpwn · 2014-02-04 22:41:47

DrZaius wrote:

Maybe not for you, but for your users if they want to download their converted file and then use it in a fashion that would benefit from having the moov atom at the beginning of the file.

Fair enough. I'll do some testing with this.

DrZaius wrote:

-crf 18 might be considered overkill. The default (-crf 23) would probably be ~30-40% smaller in size (depending on the input) and most users probably won't notice a difference. A faster preset might not hurt either. Users are impatient.

I spent a lot of time (on the scale of hours) with lots of input files and lots of tests finding the right quality, and tweaking the settings for playback on all browsers/platforms/devices. I settled on -crf 18, because I often saw noticable quality loss with higher values. We have some sexy encoding machines - we're encoding on 8 cores most of the time. It doesn't take too long.

DrZaius · 2014-02-04 22:50:04

SirCmpwn wrote:

I settled on -crf 18, because I often saw noticable quality loss with higher values. We have some sexy encoding machines - we're encoding on 8 cores most of the time. It doesn't take too long.

Sounds good to me then. You just may want to keep it in mind if you get too popular.

Looks like a fun project. I'll make sure to point it out to others looking for this type of service.

likytau · 2014-02-05 05:41:24

SirCmpwn wrote:

likytau wrote:
a GIF (we convert those to video, makes them load faster)
practice, but of course it totally breaks on pixel art.
You're absolutely right. The reason we decided to convert GIF to video is because the majority of GIFs are sourced from video, and GIF is a poor choice for that kind of media. However, GIF is well suited to pixel art. The original GIF doesn't go away when you upload - if you want to host GIFs on MediaCrush like that, I suggest hotlinking straight to it: https://mediacru.sh/1hBjPzWZGy6m.gif. That being said, we strive to make the conversion as close as possible to the original, and I feel that in both cases, the final result is close enough to give the general idea, and then you get the original GIF when you click "Download", so it's suitable for sharing.

Yes, I agree that it is quite good quality, the problem only arises when the output must be 100% accurate as with pixel art.
It's good to know that the original does not go away.. Especially for formats like GPL, this is quite helpful.

Files are run through detect, which determines what kind of file they are, collects some metadata, and selects a processor. Processing is distributed, which is accomplished through Celery (with Redis as a broker).

I see, excellent summary. With the help of that, I have written a GPL detector (which maybe doesn't work fully, I had no idea whether simply open()ing the file would be okay with your architecture):

def detect_gpl(path):
    # XXX don't know how to access path. Is standard open() ok?
    #
    with open(path, 'r') as f:
        lines = [f.readline().rstrip() for v in range(3)]
        if (lines[0] != 'GIMP Palette' or 
            (not lines[1].startswith('Name: ')) or
            (not lines[2].startswith('Columns: '))):
            return None
    return {
        'type': 'application/x-gimp-palette',
        'extra': None,
        'flags': None,
    }

Haven't written rendering code yet, but I think imagemagick can do the job; it would probably be possible to generate an XPM directly in Python code, that ImageMagick could then convert.

Regarding the other formats you mentioned, could you offer some test files, and a general idea of what they are and how we might go about detecting them and supporting them?

Yeah, I'll provide some shortly. You may not want to support them all, but here's a summary of what they are:

* GPL : Gimp Palette. Used by GIMP (duh), Inkscape, Mypaint, Scribus ... most OSS graphics apps. Identified as above (well, my routine above is actually fairly paranoid -- just checking that the first line content equals 'GIMP Palette' is normally enough to detect it). Sample files here.
* ora: OpenRaster. 'Generic layered image format', hopefully will become a standard for layered raster graphics interchange. MyPaint's native format, supported by GIMP and Krita. Convertable to PNG using calligraconverter[1].
* kra: Krita's native format. Convertable to PNG using calligraconverter[1].
* xcf, xcf.gz, xcf.bz: GIMP's native format. Convertable to PNG using xcf2png (from 'xcftools' package)
* ase : probably don't bother -- this is Aseprite's native format, which is not particularly common in the wild. I don't know of a tool that converts it. I just tried it for testing's sake.
* webp: Google's WebP format. Already handled, but you seem to currently convert it to a video even if the image is static.

[1] Available in the package 'calligra-extras'

You're welcome to submit pull requests; you should join the IRC channel to participate in dev chatter. Thanks for the feedback!

Thanks, I'll look at doing that once I'm familiar with the system.

Last edited by likytau (2014-02-05 05:45:51)

SirCmpwn · 2014-02-05 06:16:04

I've pushed an update that handles webp and xcf. I'm hesitant to support the others just yet because of the bulky deps - takes longer to do a security audit on them.

Last edited by SirCmpwn (2014-02-05 06:19:40)

orschiro · 2014-02-05 08:53:54

Sounds cool!

I just want to wish you the best with your project and thanks a bunch for this sophisticated contribution to FOSS!

Last edited by orschiro (2014-02-05 08:54:12)

SirCmpwn · 2014-02-05 10:07:03

Hey DrZaius, I've updated it to use jpegtran for jpeg optimization (thanks, it works really well), and I've made libfdk_aac explicit, and specified -q:v for the theora video quality. Thanks for the tips!

orschiro, glad you like it! I saw your GitHub issue on the Chrome plugin, I'll see if I can do that myself or find someone to do it soon.

rtfreedman · 2014-02-05 22:27:32

Whois: mediacru.sh

Domain : mediacru.sh
Status : Live
Expiry : 2014-07-22

Somehow, I don't trust you - 6 month to go before closing the honey-pot?

SirCmpwn · 2014-02-05 22:29:33

If you don't trust me, install it on your own server. The readme offers detailed instructions for installing it locally or on a dedicated server. You don't have to use the public site if you don't want to. That being said, we had no clue if this would take off (it did), so we only bought the domain for a year. We will renew it, of course, when the time comes.

rtfreedman · 2014-02-05 22:38:10

If you don't trust me, ...

Sorry if I misjudged your project - I'll have a closer look tomorrow.
Maybe I get paranoid these days ....

Last edited by rtfreedman (2014-02-05 22:38:43)

Kolibry · 2014-02-06 08:02:57

Hey, your project looks really interesting, thanks for sharing !

Works very wel and is user-friendly even if I just tested some basic stuff. I love the options, especially a bonus to be able to switch to dark theme, every sites should do so aha

SirCmpwn · 2014-02-06 13:27:16

Just pushed another update and I thought I'd let you guys know. The processing backend has been extended to support additional metadata, and all metadata has been exposed via the public API. This change lets us do things like show song titles. Here are some examples of what you can get from the API now:

sircmpwn@homura ~> curl https://mediacru.sh/IFG_FP1L1zti.json # Full API docs at https://mediacru.sh/docs
{
  "blob_type": "audio", 
  ...
  "metadata": {
    "album": "Mahou Shoujo Madoka Magica Original Soundtrack 2", 
    "artist": "Yuki Kajiura", 
    "duration": 288.506667, 
    "title": "Ave Maria"
  }, 
}
sircmpwn@homura ~> curl https://mediacru.sh/JfTV-R-eg_2Z.json
{
  "blob_type": "image", 
  ...
  "metadata": {
    "dimensions": {
      "height": 750, 
      "width": 980
    }
  }, 
}
sircmpwn@homura ~> curl https://mediacru.sh/Cjbft93YWNNu.json
{
  "blob_type": "video", 
  ...
  "metadata": {
    "dimensions": {
      "height": 720, 
      "width": 1280
    }, 
    "duration": 90.025833, 
    "has_audio": true, 
    "has_video": true
  },
}

The changes went down in this pull request if you want to see the code involved. If you were running your own instance and you want to crawl existing files for metadata, run migrate.py after you update. Feedback is very welcome!

sekret · 2014-02-09 10:54:07

Great project! I'll definitely use it starting right now.

SirCmpwn · 2014-03-08 03:05:03

Stopping by again to mention a recent update: https://blog.mediacru.sh/2014/03/01/Add … Crush.html

Support for subtitles! Here's the pull request it happened in: https://github.com/MediaCrush/MediaCrush/pull/513

Last edited by SirCmpwn (2014-03-08 03:05:25)

Arch Linux

#1 2014-02-04 01:22:04

Open source image, audio, video hosting - MediaCrush

#2 2014-02-04 14:14:37

Re: Open source image, audio, video hosting - MediaCrush

#3 2014-02-04 14:33:12

Re: Open source image, audio, video hosting - MediaCrush

#4 2014-02-04 19:07:19

Re: Open source image, audio, video hosting - MediaCrush

#5 2014-02-04 19:15:47

Re: Open source image, audio, video hosting - MediaCrush

#6 2014-02-04 19:21:29

Re: Open source image, audio, video hosting - MediaCrush

#7 2014-02-04 19:33:22

Re: Open source image, audio, video hosting - MediaCrush

#8 2014-02-04 22:17:45

Re: Open source image, audio, video hosting - MediaCrush

#9 2014-02-04 22:21:00

Re: Open source image, audio, video hosting - MediaCrush

#10 2014-02-04 22:38:09

Re: Open source image, audio, video hosting - MediaCrush

#11 2014-02-04 22:41:47

Re: Open source image, audio, video hosting - MediaCrush

#12 2014-02-04 22:50:04

Re: Open source image, audio, video hosting - MediaCrush

#13 2014-02-05 05:41:24

Re: Open source image, audio, video hosting - MediaCrush

#14 2014-02-05 06:16:04

Re: Open source image, audio, video hosting - MediaCrush

#15 2014-02-05 08:53:54

Re: Open source image, audio, video hosting - MediaCrush

#16 2014-02-05 10:07:03

Re: Open source image, audio, video hosting - MediaCrush

#17 2014-02-05 22:27:32

Re: Open source image, audio, video hosting - MediaCrush

#18 2014-02-05 22:29:33

Re: Open source image, audio, video hosting - MediaCrush

#19 2014-02-05 22:38:10

Re: Open source image, audio, video hosting - MediaCrush

#20 2014-02-06 08:02:57

Re: Open source image, audio, video hosting - MediaCrush

#21 2014-02-06 13:27:16

Re: Open source image, audio, video hosting - MediaCrush

#22 2014-02-09 10:54:07

Re: Open source image, audio, video hosting - MediaCrush

#23 2014-03-08 03:05:03

Re: Open source image, audio, video hosting - MediaCrush

Board footer