337 Commits

Author SHA1 Message Date
Christopher Allan Webber
79f7cfd900 Merge remote-tracking branch 'refs/remotes/spaetz/521_license_preference' into mergetest 2013-01-22 13:43:02 -06:00
Elrond
afbfd40578 Enable foreign key checking on sqlite.
sqlite normally does not check foreign key referential
integrity. But it can do so.
So let's use it. Better safe than sorry.
2013-01-18 18:42:52 +01:00
Mark Holmquist
dc4dfbde35 Add a license preference field
This feature is absolutely necessary. Now a user can simply define
their default license and quickly go through a form, as opposed to
stopping to click on the select and choosing the same option over
and over again.

Also added DB migration for the field, so that's working now, too.

Rebased by Sebastian and made the default value to be unicode.

Reviewed-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-17 21:34:04 +01:00
Sebastian Spaeth
3809a8b8e2 import db.sql.util -> db.util
Merging an old branch, I reintroduced an import of db.sql.util rather than
db.util. Fixing the glitch.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-17 12:19:33 +01:00
Sebastian Spaeth
03b4fc500c Implement User.delete() (#540)
Set User.collections to her Collections using the backref feature.
This way we can iterate a user's collections and delete them all.

Delete all MediaEntries/Files/attachments/comments/collections etc
before finally deleting the User object. This is the backend work for
issue 302 (allow a user to delete ones own account)
2013-01-17 11:48:49 +01:00
Sebastian Spaeth
242776e363 Implement Collection.delete()
Deleting a Collection should automatically delete all containing items.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-17 11:48:49 +01:00
Sebastian Spaeth
fdc34b8ba7 Implement MediaEntry().delete() (#540)
Deleting a MediaEntry instance will automatically
delete all related comments and files/attachments. This moves
implementation logic out of views.py and allows to make use of this
functionality when e.g. deleting a User() account.

Whenever a MediaEntry entry is deleted, this will also sql-delete
the corresponding MediaFile entry.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-17 11:48:07 +01:00
Sebastian Spaeth
65969d3fb7 Simplify check_media_slug_used
Remove the unused dummy_db argument and generally make the function
readable.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-15 14:49:50 +01:00
Elrond
c130e3ee79 Move db.sql.migration_tools to db.migration_tools.
Follow the new trend.
2013-01-08 22:50:01 +01:00
Elrond
c4466cb4dc Add inspect_table.
In our current scheme of migrations, we reflect the current
sql schema into an SQLAlchemy schema. So let's have a tool
function for this.
2013-01-08 22:50:01 +01:00
Elrond
a050e776c6 Move all the migration tools into new migration_tools.py
Factor all the migration related stuff out into a new
.db.sql.migration_tools.
First we don't have to load this module for our normal
server.
Second it makes all the import dependencies a little more
cleaner.
2013-01-08 22:50:01 +01:00
Sebastian Spaeth
20be9bb77d Remove custom GMQuery class
We provided a custom GMQuery class that offered a .sort() method for
compatibility with the Mongo syntax. Now that we have settled for sqlalchemy
which uses the order_by() method, we can safely remove this custom class
and move a little closer to "pure" and native sqlalchemy usage.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-07 13:52:37 +01:00
Sebastian Spaeth
39dc3bf8db Mv db.sql.base to db.base
This concludes the db.sql.* -> db.* move. Our db abstraction layer is
sqlalchemy, so there is no need to a separate db.sql.* hierarchy.

All tests have been run for each of the commit series to make sure
everything works at every step.
2013-01-07 13:42:32 +01:00
Sebastian Spaeth
a5acfe23fa Move mediagoblin.db.sql.extratypes to mediagoblin.db.extratypes
No other functional changes.
2013-01-07 13:10:02 +01:00
Sebastian Spaeth
b0c8328e54 Move db.sql.models* to db.models* 2013-01-07 11:44:29 +01:00
Sebastian Spaeth
1e46dc2537 Move db.sql.util to db.util
Now that sqlalchemy is providing the database abstractions, there is no
need to hide everything in db.sql. sub-modules. It complicates the code
and provides a futher layer of indirection.

Move the db.sql.util.py to db.util.py and adapt the importers.
2013-01-07 11:31:56 +01:00
Sebastian Spaeth
0efe9e2796 Remove mediagoblin.db.sql.fake.DESCENDING
This is the last remnant that requires us to keep db.sql.fake.py. Use
ModelName.desc() or sqlalchemy.sql.expression.desc(column) to achieve
descending sorts.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2013-01-07 11:15:04 +01:00
Sebastian Spaeth
71717fd531 Remove ObjectId from the tree
This was one of the last remaining Mongo holdouts and has been removed from
the tree herewith. Good bye, ObjectId.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-25 20:52:25 +01:00
Sebastian Spaeth
7c029a1f33 Remove InvalidId
It was a NoOp in our Non-mongo world. So it is safe to remove.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-25 20:28:19 +01:00
Sebastian Spaeth
9536988425 Move db.sql.migrations to db.migrations 2012-12-25 20:12:16 +01:00
Sebastian Spaeth
6eddc3b75e Move db.sql.open to db.open
Now that mongo has been ripped out and sqlalchemy is already providing
the database abstraction, there is no need to hide everything in the sql
module. Transition db.sql.open to db.open and adapt all direct importers.
2012-12-25 20:09:43 +01:00
Sebastian Spaeth
bc142abc55 RIP out mongo
Since sqlalchemy is providing our database abstraction and we have
moved away from Mongo as the underlying database, it is now time to
simplify things and rip out mongo. This provides the bulk of the
changes, and can stand on its own. There are some followup tasks
that can be done, such as removing now unneeded abstraction layers,
e.g. db.sql.fake.py
2012-12-25 20:06:03 +01:00
Elrond
b39d1f2351 Mongo removal: Remove the validate=True arg to obj.save()
all callers were forced to use validate=True anyway. So
remove this useless stuff.
2012-12-24 00:07:24 +01:00
Sebastian Spaeth
10f1f2f56b Remove Mongoism query.skip()
sqlalchemy supports slice() or [n:m] just fine.

Right now, it seems we cannot distinguish beween "empty" results
and out-of bound slices. It would be nice if we could distinguish
these somehow.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-21 11:30:29 +01:00
Sebastian Spaeth
30bb4109bc Remove SimpleFieldAlias
It was only used for the model._id -> model.id conversion and is not
needed anymore.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-21 00:30:48 +01:00
Sebastian Spaeth
5c2b84869f Move DBModel._id -> DBModel.id
We were refering to model._id in most of the code base as this is
what Mongo uses. However, each use of _id required a) fixup of queries:
e.g. what we did in our find() and find_one() functions moving all
'_id' to 'id'. It also required using AliasFields to make the ._id
attribute available. This all means lots of superfluous fixing and
transitioning in a SQL world.

It will also not work in the long run. Much newer code already refers
to the objects by model.id (e.g. in the oauth plugin), which will break
with Mongo. So let's be honest, rip out the _id mongoism and live with
.id as the one canonical way to address objects.

This commit modifies all users and providers of model._id to use
model.id instead. This patch works with or without Mongo removed first,
but will break Mongo usage (even more than before)

I have not bothered to fixup db.mongo.* and db.sql.convert
(which converts from Mongo to SQL)

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-21 00:30:48 +01:00
Christopher Allan Webber
78fd5581a9 Remove print statments from this migration.
The reason for wanting to give extra information to the user (this is
a very special case migration) is good, but we don't have a nice
"official" way to capture and present that information during tests,
so removing this.
2012-12-10 16:17:24 -06:00
Elrond
a64abbb10a Fix typo, disable debugging. 2012-12-10 16:17:24 -06:00
Elrond
ea91c183ff UniqueConstraing migration: Adding the explaining comments.
Add a lengthy comment explaining all the variants.
2012-12-10 16:17:24 -06:00
Elrond
0f14c362c3 Fixing our broken CollectionItem unique constraint.
This one seems to work nicely in all relevant situations.
See comments inside the source.
2012-12-10 16:17:24 -06:00
Nathan Yergler
afe0afdb88 Ensure query_dict is a dict after the contents have been modified.
_fix_query_dict modifies its argument in place. Ensure that the
argument passed in has a local name and will be passed into the
subsequent filter_by call.
2012-12-10 12:30:04 +01:00
Christopher Allan Webber
41fc4698c5 Eeek! Set the migration number to the current migration number being run! 2012-12-09 12:20:06 -06:00
Christopher Allan Webber
a4eef7fe91 Elrond's suggestion: have set_current_migration execute after each migration run. 2012-12-09 12:12:07 -06:00
Sebastian Spaeth
5f8b4ae895 make media_manager a property of MediaEntry in mixin.py
In all cases where get_media_manager(_media_type_as_string) was called in
our code base we ultimately passed in a "MediaEntry().media_type" to get
the matching MEDIA_MANAGER. It so makes sense to make this a function of
the MediaEntry rather than a global function in mediagoblin.media_types and
passing around media_entry.media_type as arguments all the time.

It saves a few import statements and arguments. I also made it so the
Media_manager property is cached for subsequent calls, although I am not too
sure that this is needed (there are other cases for which this would make
more sense)

Also add a get_media_manager test to the media submission tests. It submits
an image and checks that both media.media_type and media.media_manager
return the right thing. Not sure if these tests could not be merged with an
existing submission test, but it can't hurt to have things explicit.

TODO: Right now we iterate through all existing media_managers to find the
right one based on the string of its module name. This should be made a simple
dict lookup to avoid all the extra work.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-12-04 15:15:41 +01:00
Sebastian Spaeth
138a18fd6e Implement licenses.get_license_by_url
Rather than exploding in the user's face (for example if we custom-configure
licenses in our MG instance, and there are still media with now "unknown"
licenses in the db), simply return a License object as a fallback, where all
attributes are set to the URL we were handed.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-11-30 14:31:43 +01:00
Sebastian Spaeth
df1c497609 Use mediagoblin import machinery
Rather than manually __importing__ the MEDIA_MANAGER, we should have
been using tools.common.import_component in the first place.

But even better to use the existing get_media_manager() function that
exists for exactly our purpose. Also improve documentation of what happens
in case of failure.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-11-30 14:25:26 +01:00
Sebastian Spaeth
27886480ab Remove importlib (python2.7'ism)
importlib is cool, but only included in python2.7 which is beyond our
minimum python version that we support. So simply use plain old
__import__.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-11-30 11:25:50 +01:00
Elrond
efb70b529f Merge some small bits by spaetz. Needed later.
* commit '9437ea4742':
  Add commit argument to clean_orphan_tags
  Add "commit" argument to Base model delete()
2012-11-28 16:45:15 +01:00
Sebastian Spaeth
9437ea4742 Add commit argument to clean_orphan_tags
So we can prevent the session from being committed if we don't want it.
2012-11-28 14:43:04 +01:00
Elrond
329e39034b Add "commit" argument to Base model delete()
In case we want to bundle db actions into a single transaction, we
can now use delete(commit=False) to prevent the transaction from being
committed immediately. This is useful when e.g. deleting a User() and
thousands of his MediaEntries in a single commit.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-11-28 13:52:41 +01:00
Christopher Allan Webber
fbfcf1a57a Merge remote-tracking branch 'refs/remotes/elrond/sql/migrate_new_tables' 2012-11-26 15:15:13 -06:00
Elrond
9abd664bf2 Stop unicode conversion warning during dbupdate.
Just force the passed in name to MigrationManager() to unicode.
2012-11-24 22:57:46 +01:00
Elrond
316e1dfdde SQL Migrations: Rewrite table creation completely.
We have migrations creating new tables. Those currently use
"raw" table definitions. This easily gives errors (we
already had this problem).

So instead rewrite those to use declarative tables and use
those to create new tables. Just copy the new table over to
the migration, strip it down to the bare minimum, rename to
_v0, base it on declarative_base() and be done!

Do this for the current migrations.
2012-11-24 19:23:08 +01:00
Sebastian Spaeth
2e4ad35962 Simplify/Robustify the thumbnail URL usage in templates
So far templates required a very complex blurb to simply insert a
thumbnail URL, exposing much of the internal logic to the template
designer. In addition, we would fail with an error if for some
reason the media_files['thumb'] entry was never populated.

This adds the MediaEntry.thumb_url property that template designers
can simply use. It will do the right thing, either fetching the proper
thumbnail or hand back a generic icon specified in a media's
MEDIA_MANAGER as "default_thumb".

Add an image default fallback icon (stolen from Tangos, which are
Public Domain since version 0.8.90 as I understand) since the one
we referred to was not existing. Perhaps, a "broken image" icon
would be better, but I'll leave that to our capable designers.

All templates have been modified to make use of the new thumb_url
function.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2012-11-15 12:01:31 +01:00
Joar Wandborg
942084fbfd Added cascade all, delete-orphan to ProcessingMetaData 2012-09-27 11:59:17 +02:00
Joar Wandborg
939d57a022 HTTP callback fixes
- Added HTTPError catching around the callback request, to not mark the
  entry as failed, just log the exception.
- Fixed bug where I forgot to actually fetch the entry before passing it
  to json_processing_callback.
- Changed __main__ migration #6 to create the ProcessingMetaData table
  as it is currently, to prevent possible breakage if a siteadmin
  is lagging behind with his db migrations and more than one migration
  wants to fix stuff with the ProcessingMetaData table.
2012-09-26 23:53:51 +02:00
Joar Wandborg
5354f954dc Added support for http callbacks on processing
Sends an HTTP POST request back to an URL given on submission to the API
submit view.
2012-09-26 23:53:51 +02:00
Joar Wandborg
88a9662be4 Added client registration caps to OAuth plugin
THE MIGRATIONS SUPPLIED WITH THIS COMMIT WILL DROP AND RE-CREATE YOUR
oauth__tokens AND oauth__codes TABLES. ALL YOUR OAUTH CODES AND TOKENS
WILL BE LOST.

- Fixed pylint issues in db/sql/migrations.
- Added __repr__ to the User model.
- Added _disable_cors option to json_response.
- Added crude error handling to the api.tools.api_auth decorator
- Updated the OAuth README.
- Added client registration, client overview, connection overview,
  client authorization views and templates.
- Added error handling to the OAuthAuth Auth object.
- Added AuthorizationForm, ClientRegistrationForm in oauth/forms.
- Added migrations for OAuth, added client registration migration.
- Added OAuthClient, OAuthUserClient models.
- Added oauth/tools with require_client_auth decorator method.
2012-09-21 13:09:42 +02:00
Joar Wandborg
6d1e55b262 Style fixes 2012-09-18 18:19:26 +02:00
Aaron Williamson
256f816f50 Fixed collection sidebar for media_home to user collection.url_for_self instead of incorrectly getting the session user 2012-09-18 18:10:36 +02:00