Factor all the migration related stuff out into a new
.db.sql.migration_tools.
First we don't have to load this module for our normal
server.
Second it makes all the import dependencies a little more
cleaner.
We copy uploaded media from the queue store to the local workbench
and then to its final destination. The latter was done by simply:
dst.write(src.read()) which is of course evil as it reads the whole
file content into RAM. Which *might* arguably still be OK for
images, but you never know.
Make use of the provided storage() methods that offer chunked copying
rather than opening and fudging with files ourselves.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
These tests, don't need fresh databases, so don't discard and recreate
the tables. This reduces test suite runtime on my laptop from 130 to 96
seconds.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When the tests want to create a new user, don't fail if it already
exists and just reuse the existing one. This allows us to run tests
without dumping the whole database if that is not needed for the tests.
The upcoming tests for test_edit will make use of this.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Removing the Mongo InvalidID legacy code removed an explicit check for
"int" for the id lookup. This led the @get_user_media_entry decorator to
fail if we looked up a nonexisting non-numerical slug (it tried to query
the id with a string, which failed). Cast id to int and return 404 in
case it is non-numeric which fixes the regression. It does not fix the
underlying problem of slug_or_id lookups that were discussed.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We provided a custom GMQuery class that offered a .sort() method for
compatibility with the Mongo syntax. Now that we have settled for sqlalchemy
which uses the order_by() method, we can safely remove this custom class
and move a little closer to "pure" and native sqlalchemy usage.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This concludes the db.sql.* -> db.* move. Our db abstraction layer is
sqlalchemy, so there is no need to a separate db.sql.* hierarchy.
All tests have been run for each of the commit series to make sure
everything works at every step.
Now that sqlalchemy is providing the database abstractions, there is no
need to hide everything in db.sql. sub-modules. It complicates the code
and provides a futher layer of indirection.
Move the db.sql.util.py to db.util.py and adapt the importers.
This is the last remnant that requires us to keep db.sql.fake.py. Use
ModelName.desc() or sqlalchemy.sql.expression.desc(column) to achieve
descending sorts.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
First rename prepare_entry to prepare_queue_task, because
this is really more like what this thing does.
Thanks to Velmont for noting that we do not need a request
in here, but an "app" is good enough. Which means, that
this stuff can be called from tool scripts too.
Previous exif bump copied the newer version into
mediagoblin/tools/extlib/exif but not in extlib/exif. Fix this by
bumping extlib/exif and symlinking to tools/extlib/exif.
Do note that this is still the version fetched from http://sourceforge.net/projects/exif-py/
while the upstream maintainer seems to be active on:
https://github.com/ianare/exif-py
(The sf.net download is quite new though)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This was one of the last remaining Mongo holdouts and has been removed from
the tree herewith. Good bye, ObjectId.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Now that mongo has been ripped out and sqlalchemy is already providing
the database abstraction, there is no need to hide everything in the sql
module. Transition db.sql.open to db.open and adapt all direct importers.
Since sqlalchemy is providing our database abstraction and we have
moved away from Mongo as the underlying database, it is now time to
simplify things and rip out mongo. This provides the bulk of the
changes, and can stand on its own. There are some followup tasks
that can be done, such as removing now unneeded abstraction layers,
e.g. db.sql.fake.py
Bump bundled EXIF lib to 1.0.10 as release in Sep 2012.
Also skip unused detailed EXIF tags for reading, we might turn that
on when we need them.
Adapt test to wording change in EXIF Flas field and due to the fact
that we use "details=False" by default now (we did not use these
anyway)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
There is nothing inherently wrong with the currently bundled version,
but the last one was uploaded in March 2012 and there have been
upstream releases since. So bump to the latest available release 3.2.3.
Might help with the reported issue 512 of an unresponsive video player.
Do note that the Flash fallback option is removed. If we decide
this should be added it is easy to add it back.
We still use our own customized theme.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We need to know the name of the backref, so that we can
access it by name on the MediaEntry. We might be able to
get this name by inspection, but this way is easier, for
now.
We should handle SQLAlchemy warnings. And to make that a
lot easier, turn them into real errors. That way they
1) Turn up more prominently.
2) Have a useful backtrace.
This only happens in the testsuite, so that normal useage
is not impacted.
MGRoute subclasses Rule():
Rule doesn't have a way to tag extra data, like the
controller function, we need. So MGRoute has a new
attribute .gmg_controller, which holds this.
Rewrite everything to use this new Rule variant and drop
all the other stuff that mapped endpoints to controller
functions, mostly.
Move most of the "init routing" code inside get_url_map().
Maybe not the best solution, but better than running the
init code directly at module load time.
Elrond correctly remarked that we should be comparing user by id
and not by comparing User objects (as I mistakenly did). He is
right, of course!
Also removing the 2 stray debug prints that were left over.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Thanks to Elrond's review. 1) Use filter_by more consistently
rather than mixing filter and filter_by. 2) Add multiple AND
conditions in the same filter rule rather than using separate
ones. 3) The atom feeds used filter_by(Modelname.attr == ...)
which was the wrong pattern, of course.
Thanks for repairing my junk Elrond!
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
