148 Commits

Author SHA1 Message Date
Christopher Allan Webber
d0ceb506bd Fixing dates when uploaded in a video to a mediagoblin instance.
There were some "serializing to json strings" issues.  They should be
fixed now... much more careful whitelist and cleaning of the video
"tags" metadata out of gstreamer.

This commit sponsored by Aimee Sullivan.  Thanks!
2013-03-11 16:54:41 -05:00
Christopher Allan Webber
7bf229267d Making a fix so that video codec name switched from "vp8 video" to "vp8"
"vp8 video" is what vp8 is marked as in gstreamer's metadata.
However, the browser expects it just as the name "vp8".  So fixing
that.

This commit sponsored by Tyng-Ruey Chuang.  Thank you!
2013-03-06 12:05:40 -06:00
Christopher Allan Webber
f51a416778 Removing an unnecessary video write
In the case of if we're skipping transcoding, we don't need to copy
this file at all!

This commit sponsored by Frank Zambrini III.  Thanks!
2013-03-04 11:53:04 -06:00
Christopher Allan Webber
4f239ff194 Another elrond suggestion: only init orig_metadata if there's anything in the dict.
This commit sponsored by Joshua Rosen.  Thank you!
2013-03-03 14:29:30 -06:00
Christopher Allan Webber
9d4cebfd36 Per Elrond's suggestions moving DEFAULT_WEBM_TYPE to media manager
The reason for this is to avoid defining this twice as we were
previously (once in the template, once in video/models.py)

This commit sponsored by Roland McIntosh.  Thank you!
2013-03-03 14:27:36 -06:00
Christopher Allan Webber
787aafd644 Extrapolate type= based on the video metadata that we have, if we can.
It's kind of awkward because it relies on there being a entry.media_data,
but that's not guaranteed... (see http://issues.mediagoblin.org/ticket/650)
so we use a dopey fallback in the template in that case (kind of
annoying info duplication).

This commit sponsored by Piotr Wieczorek.  Thank you!
2013-03-03 11:41:40 -06:00
Christopher Allan Webber
3ff006ef3b pdb.set_trace() in mediagoblin code is Not Allowed(TM), removing from audio code
Of course, the version that appears here is not really dangerous
because it's for the "call the file individually" form of debugging,
but it isn't allowed anyway.

This commit sponsored by Michael Faryniarz.  Thanks!
2013-03-03 10:38:06 -06:00
Christopher Allan Webber
29adab4651 Now store metadata info from processing into the media type.
This comes in several parts:
 - Store the metadata from gstreamer during processing
 - Add a new JSONEncoded field to the VideoData table
 - And, of course, add a migration for that field!

This commit sponsored by Julius Tuomisto.  Thank you, Julius!
2013-03-03 10:36:37 -06:00
Christopher Allan Webber
ddbf6af1e2 Huge amount of work to (mostly) allow .ogg (and maybe other) formats to skip transcode
- Update get_display_media in several ways:
   - now uses the media type's own declaration of the order of things
   - returns both the media_size and the media_path, as per the docstring
   - implicitly uses self.media_files as opposed to forcing you to pass it in
 - update videos to use get_display_media
 - update images to declare media_fetch_order in the media manager (videos also)
 - update stl to use media.media_files['original'] instead of weird
   use of get_display_media
 - update sidebar to only conditionally show webm_640

TODO still: identify video type information *during* processing, show
that in the <video><source /></video> element.

This commit sponsored by Nathan Yergler.  Thanks, nyergler!
2013-03-02 19:06:31 -06:00
Christopher Allan Webber
fd693e368b If we're not transcoding, copy this file directly over to ['original']
There's no reason to copy it over to 'webm_640' in such a case,
clearly.

Added logic so we don't do it twice either.

Haven't tested this yet though ;)

This commit sponsored by Algot Runeman.  Thank you!
2013-03-01 17:28:07 -06:00
Christopher Allan Webber
ed6344009a Merge branch 'master' into joar-skip_transcoding
Conflicts:
	mediagoblin/config_spec.ini
2013-03-01 16:29:22 -06:00
Joar Wandborg
6bdf9a1044 Allow media managers without sniff_handler 2013-02-27 00:30:51 +01:00
Joar Wandborg
8d355df617 Tuned logging and added comments to 511 fix 2013-02-10 20:19:13 +01:00
pythonsnake
34c35c8cec Fixed issue #511. 2013-02-10 20:11:47 +01:00
Christopher Allan Webber
591c116098 Merge remote-tracking branch 'refs/remotes/elrond/sql/media_data' 2013-02-09 15:33:06 -06:00
Elrond
fb46fa663d Kill monkeypatching of ProcessingState.
And change the process_foo() API to accept a
processingstate now.

image and video are tested, the others are UNTESTED.
2013-02-08 10:09:37 +01:00
Elrond
715ea49546 Also refactor "copy original into public storage".
This makes the processing code easier to read/write and
alos will help the reprocessing once we get to it.

Thanks to Joar Wandborg for testing!
2013-02-08 10:09:37 +01:00
Elrond
bfd68cce85 Use ProcessingState for video.
Rewrite video to use the new ProcessingState.

Thanks to Joar Wandborg for testing!
2013-02-08 10:09:37 +01:00
Elrond
93b14fc300 Implement ProcessingState class and use for images
The idea is to have a class that has the knowledge of the
currently being processed media and also has tools for
that.

The long term idea is to make reprocessing easier by for
example hiding the way the original comes into the
processing code.
2013-02-08 10:05:42 +01:00
Elrond
1f268deda4 Turn media.{backref_nam} from a list to a scalar.
For all our media_types, let the backref on the media_entry
be a scalar (there is only one media_data per media_entry)
instead of a list with zero or one entry.

The media_data toolchain on MediaEntry currently handles
both transparently.
2013-02-01 22:02:35 +01:00
Joar Wandborg
d63cc34e71 Made pil_filters a PIL_FILTERS (global) 2013-01-22 23:28:17 +01:00
Joar Wandborg
7cd7db5af4 Made the image sampling filter configurable
- Changed the default to BICUBIC instead of previous ANTIALIAS
2013-01-22 23:28:17 +01:00
Joar Wandborg
5c754fdaee Added option to skip transcoding
- If the video input matches the configurable rules, just copy it to the
  output without transcoding it.
2013-01-22 23:03:14 +01:00
Sebastian Spaeth
c00e18fe49 Do not read complete videos in RAM (#419)
We were reading the complete "medium" "thumbnail" and "original"
in RAM via dst.write(src.read()). Just call the appropriate storage
methods copy_local_to_storage which are responsible for streaming
local files efficiently.

The efficiency of this patch depends on the separate branch that
actually implements chunked copying for Storage().copy_local_to_storage()
2013-01-16 20:19:26 +01:00
Sebastian Spaeth
45ab3e07ef Switch over media processor to use the get_workbench decorator (#565)
This makes workbench getting more convenient by eliminating some
boilerplate and more robust by cleaning the workbench up even if processing
ends with an Exception.

Finally, this fixes the bugs in the ascii and video backends to never call
workbench.destroy, so those workbenches were never cleaned up.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-16 20:19:23 +01:00
Christopher Allan Webber
f26c097d3e Make the workbench destroy itself for video processing also. 2013-01-16 12:12:57 -06:00
Christopher Allan Webber
3c5f583707 Merge remote-tracking branch 'refs/remotes/spaetz/561_use_workbench_not_tempfiles' 2013-01-16 11:23:38 -06:00
Christopher Allan Webber
8e7354ef2b Fixing import to mediagoblin.db.base in stl models
Previously we had mediagoblin.db.sql.base, that just changed to drop
the sql; fixing.
2013-01-08 13:24:27 -06:00
Sebastian Spaeth
25e398428b Audio and video should use workbench instead of tempfiles (#561)
We were using lots of tempfiles in the audio and video processing
backends which worked around our workbench system. Still use the
tempfiles package but create them in the workbench directory. This
can help address the uploads of large files (#419) where /tmp might
be a smallish tmpfs and our workbench a real disk.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-08 15:05:03 +01:00
Sebastian Spaeth
5018a3557c Don't read full image media into RAM on copying (#419)
We copy uploaded media from the queue store to the local workbench
and then to its final destination. The latter was done by simply:
dst.write(src.read()) which is of course evil as it reads the whole
file content into RAM. Which *might* arguably still be OK for
images, but you never know.

Make use of the provided storage() methods that offer chunked copying
rather than opening and fudging with files ourselves.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-08 14:51:41 +01:00
Sebastian Spaeth
39dc3bf8db Mv db.sql.base to db.base
This concludes the db.sql.* -> db.* move. Our db abstraction layer is
sqlalchemy, so there is no need to a separate db.sql.* hierarchy.

All tests have been run for each of the commit series to make sure
everything works at every step.
2013-01-07 13:42:32 +01:00
Sebastian Spaeth
a5acfe23fa Move mediagoblin.db.sql.extratypes to mediagoblin.db.extratypes
No other functional changes.
2013-01-07 13:10:02 +01:00
Elrond
0562898e11 (SQL) Media types: Refactor backrefnames.
We need to know the name of the backref, so that we can
access it by name on the MediaEntry. We might be able to
get this name by inspection, but this way is easier, for
now.
2012-12-24 22:52:27 +01:00
Sebastian Spaeth
0efc4e4dbf Only log output and write progress to db if it has changed
De-noisify the transcoding log and db updates. Previously we would log
and save the progress percentage every second, even if it had not changed
at all. Save progress:oercentage in the Transcoder and only log/update
when the percentage has actually changed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-19 15:48:30 +01:00
Christopher Allan Webber
6af6bc05bb We don't need to save entries during processing... also adding comments explaining such 2012-12-12 10:14:24 -06:00
Sebastian Spaeth
5f8b4ae895 make media_manager a property of MediaEntry in mixin.py
In all cases where get_media_manager(_media_type_as_string) was called in
our code base we ultimately passed in a "MediaEntry().media_type" to get
the matching MEDIA_MANAGER. It so makes sense to make this a function of
the MediaEntry rather than a global function in mediagoblin.media_types and
passing around media_entry.media_type as arguments all the time.

It saves a few import statements and arguments. I also made it so the
Media_manager property is cached for subsequent calls, although I am not too
sure that this is needed (there are other cases for which this would make
more sense)

Also add a get_media_manager test to the media submission tests. It submits
an image and checks that both media.media_type and media.media_manager
return the right thing. Not sure if these tests could not be merged with an
existing submission test, but it can't hurt to have things explicit.

TODO: Right now we iterate through all existing media_managers to find the
right one based on the string of its module name. This should be made a simple
dict lookup to avoid all the extra work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-04 15:15:41 +01:00
Christopher Allan Webber
e7e435342a Switch stl processing over to using the workbench
Previously this was writing directly to the public store... that won't
work with cloudfiles backend, for example :)
2012-12-03 14:40:49 -06:00
Christopher Allan Webber
c31a501066 Correcting minor typo in exception string: Empyt->Empty 2012-12-03 14:40:49 -06:00
Christopher Allan Webber
7b8af389a9 Removing unused conversions subdir 2012-12-03 14:40:49 -06:00
Christopher Allan Webber
39c340f280 Moving blender render files to be accessed via pkg_resources
Previously the .blend and .py files had to be in the same directory
mediagoblin/celery launched from.  This is now fixed so it pulls them
out of the package proper.
2012-12-03 14:40:48 -06:00
Aeva Ntsc
de69f43aab Fixed the binary stl loader. 2012-12-03 14:40:48 -06:00
Aeva Ntsc
d25ed5dd4c Added a table to store the file extension of a model. 2012-12-03 14:40:48 -06:00
Aeva Ntsc
12a104671d Added preview rendering for a bunch of different shots. 2012-12-03 14:40:48 -06:00
Aeva Ntsc
ecee8d62c1 Catching ValueError when trying to use ObjModel on binary stl files. 2012-12-03 14:40:48 -06:00
Aeva Ntsc
76918e52e0 Added the stl/obj mediatype. 2012-12-03 14:40:47 -06:00
Sebastian Spaeth
2e4ad35962 Simplify/Robustify the thumbnail URL usage in templates
So far templates required a very complex blurb to simply insert a
thumbnail URL, exposing much of the internal logic to the template
designer. In addition, we would fail with an error if for some
reason the media_files['thumb'] entry was never populated.

This adds the MediaEntry.thumb_url property that template designers
can simply use. It will do the right thing, either fetching the proper
thumbnail or hand back a generic icon specified in a media's
MEDIA_MANAGER as "default_thumb".

Add an image default fallback icon (stolen from Tangos, which are
Public Domain since version 0.8.90 as I understand) since the one
we referred to was not existing. Perhaps, a "broken image" icon
would be better, but I'll leave that to our capable designers.

All templates have been modified to make use of the new thumb_url
function.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-11-15 12:01:31 +01:00
Joar Wandborg
e4a1b6d22a Added VideoThumbnailerMarkII
- Set video.processing to use VideoThumbnailerMarkII.
2012-10-02 16:56:29 +02:00
Joar Wandborg
f1d06e1d6c Switch from webob.Request to werkzeug.wrappers.Request 2012-09-29 21:08:20 +02:00
Joar Wandborg
4f8508f664 Removed logging entry from sniff_handler
The logging entry caused the submit page to crash if you tried to upload
an SVG with a non-ascii filename.
2012-09-22 14:19:51 +02:00
Joar Wandborg
549000d991 Audio support now creates OGG instead of WAV
While creating the spectrogram, and alternative version of the audio
file has been needed. Before this, it has been a WAV format file, the
issue with WAV is that it takes a lot of space. Starting with this it
will be an OGG file.

Rejoice :)
2012-08-12 01:57:00 +02:00