We were refering to model._id in most of the code base as this is
what Mongo uses. However, each use of _id required a) fixup of queries:
e.g. what we did in our find() and find_one() functions moving all
'_id' to 'id'. It also required using AliasFields to make the ._id
attribute available. This all means lots of superfluous fixing and
transitioning in a SQL world.
It will also not work in the long run. Much newer code already refers
to the objects by model.id (e.g. in the oauth plugin), which will break
with Mongo. So let's be honest, rip out the _id mongoism and live with
.id as the one canonical way to address objects.
This commit modifies all users and providers of model._id to use
model.id instead. This patch works with or without Mongo removed first,
but will break Mongo usage (even more than before)
I have not bothered to fixup db.mongo.* and db.sql.convert
(which converts from Mongo to SQL)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The reason for wanting to give extra information to the user (this is
a very special case migration) is good, but we don't have a nice
"official" way to capture and present that information during tests,
so removing this.
_fix_query_dict modifies its argument in place. Ensure that the
argument passed in has a local name and will be passed into the
subsequent filter_by call.
In all cases where get_media_manager(_media_type_as_string) was called in
our code base we ultimately passed in a "MediaEntry().media_type" to get
the matching MEDIA_MANAGER. It so makes sense to make this a function of
the MediaEntry rather than a global function in mediagoblin.media_types and
passing around media_entry.media_type as arguments all the time.
It saves a few import statements and arguments. I also made it so the
Media_manager property is cached for subsequent calls, although I am not too
sure that this is needed (there are other cases for which this would make
more sense)
Also add a get_media_manager test to the media submission tests. It submits
an image and checks that both media.media_type and media.media_manager
return the right thing. Not sure if these tests could not be merged with an
existing submission test, but it can't hurt to have things explicit.
TODO: Right now we iterate through all existing media_managers to find the
right one based on the string of its module name. This should be made a simple
dict lookup to avoid all the extra work.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than exploding in the user's face (for example if we custom-configure
licenses in our MG instance, and there are still media with now "unknown"
licenses in the db), simply return a License object as a fallback, where all
attributes are set to the URL we were handed.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Rather than manually __importing__ the MEDIA_MANAGER, we should have
been using tools.common.import_component in the first place.
But even better to use the existing get_media_manager() function that
exists for exactly our purpose. Also improve documentation of what happens
in case of failure.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
importlib is cool, but only included in python2.7 which is beyond our
minimum python version that we support. So simply use plain old
__import__.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
In case we want to bundle db actions into a single transaction, we
can now use delete(commit=False) to prevent the transaction from being
committed immediately. This is useful when e.g. deleting a User() and
thousands of his MediaEntries in a single commit.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We have migrations creating new tables. Those currently use
"raw" table definitions. This easily gives errors (we
already had this problem).
So instead rewrite those to use declarative tables and use
those to create new tables. Just copy the new table over to
the migration, strip it down to the bare minimum, rename to
_v0, base it on declarative_base() and be done!
Do this for the current migrations.
So far templates required a very complex blurb to simply insert a
thumbnail URL, exposing much of the internal logic to the template
designer. In addition, we would fail with an error if for some
reason the media_files['thumb'] entry was never populated.
This adds the MediaEntry.thumb_url property that template designers
can simply use. It will do the right thing, either fetching the proper
thumbnail or hand back a generic icon specified in a media's
MEDIA_MANAGER as "default_thumb".
Add an image default fallback icon (stolen from Tangos, which are
Public Domain since version 0.8.90 as I understand) since the one
we referred to was not existing. Perhaps, a "broken image" icon
would be better, but I'll leave that to our capable designers.
All templates have been modified to make use of the new thumb_url
function.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
- Added HTTPError catching around the callback request, to not mark the
entry as failed, just log the exception.
- Fixed bug where I forgot to actually fetch the entry before passing it
to json_processing_callback.
- Changed __main__ migration #6 to create the ProcessingMetaData table
as it is currently, to prevent possible breakage if a siteadmin
is lagging behind with his db migrations and more than one migration
wants to fix stuff with the ProcessingMetaData table.
THE MIGRATIONS SUPPLIED WITH THIS COMMIT WILL DROP AND RE-CREATE YOUR
oauth__tokens AND oauth__codes TABLES. ALL YOUR OAUTH CODES AND TOKENS
WILL BE LOST.
- Fixed pylint issues in db/sql/migrations.
- Added __repr__ to the User model.
- Added _disable_cors option to json_response.
- Added crude error handling to the api.tools.api_auth decorator
- Updated the OAuth README.
- Added client registration, client overview, connection overview,
client authorization views and templates.
- Added error handling to the OAuthAuth Auth object.
- Added AuthorizationForm, ClientRegistrationForm in oauth/forms.
- Added migrations for OAuth, added client registration migration.
- Added OAuthClient, OAuthUserClient models.
- Added oauth/tools with require_client_auth decorator method.
- Added progress meter for video and audio media types.
- Changed the __repr__ method of a MediaEntry to display a bit more
useful explanation.
- Added a new MediaEntry.state, 'processing', which means that the task
is running the processor on the item currently.
- Fixed some PEP8 issues in user_pages/views.py
- Fixed the ATOM TAG URI to show the correct year.
- It is now possible to actually see what's processing, due to a bug fix
where __getitem__ was called on the db model.
- Removed DEPRECATED message from the docstring, it wasn't true.
sqlite doesn't like complex changes (alter table) to happen
inside a transaction that has already done other things.
And really, each migration should say "I'm done" and commit
its changes.
This is not the full story, but it's the core of it.
Specifially the migration framework should probably do a
rollback "just in case" after each migration.
The cleanup could be missed if the request handling code in
app.py:__call__ exits early (due to exception, or due to
one of those early "return"s).
So to make sure the sql session is cleaned up for real,
wrap the whole thing in a try: finally:.
Also wrote a short tool to test if the session is actually
empty. The tool is currently disabled, but ready to be
used.
After converting everything, check what is actually used in
the db. For media_types that are not used, drop all the
media_data tables and remove the migration info.
