generate_slug could assign "none" as slug. Make sure it assigns a unique
slug in any case. We now try based on:
a) existing slug values
b) media.title
c) media.id
d) random garbage
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Previously we allowed EVERYTHING, even slashes as slug when editing the media.
Make sure we slugify the input to sanitize it.
(+ string formdata is unicode, so there is no need to convert it)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
The move to werkzeug routing went pretty smooth, but one thing was
broken by accident: URLs without final slash result in a 301
werkzeug.routing.RequestRedirect response. We displayed it as a generic
error page rather than actually sending the redirect. Do that. One thing
it does though is to skip all meddlewares, which should be OK for a 301
response, but might need rework if we decide otherwise. With this, 301
responses with lacking final slash are unbroken again.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
People(tm) want to start run_process_media from the CLI and might not
have a request object handy. So pass in the feed_url into
run_process_media rather than the request object and allow the feed url
to be empty (resulting in no PuSH notification at all then).
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Notifying the PuSH servers had 3 problems.
1) it was done immediately after sending of the processing task to celery. So if celery was run in a separate
process we would notify the PuSH servers before the new media was processed/
visible. (#436)
2) Notification code was called in submit/views.py, so submitting via the
API never resulted in notifications. (#585)
3) If Notifying the PuSH server failed, we would never retry.
The solution was to make the PuSH notification an asynchronous subtask. This
way: 1) it will only be called once async processing has finished, 2) it
is in the main processing code path, so even API calls will result in
notifications, and 3) We retry 3 times in case of failure before giving up.
If the server is in a separate process, we will wait 3x 2 minutes before
retrying the notification.
The only downside is that the celery server needs to have access to the internet
to ping the PuSH server. If that is a problem, we need to make the task belong
to a special group of celery servers that has access to the internet.
As a side effect, I believe I removed the limitation that prevented us from
upgrading celery.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Don't do "user = getUser() or newUser()" in one line. It is bound to
confuse poor souls. Be more explicit here and even add a code comment.
Thanks to Elrond for not liking the previous pattern.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We have a bunch of URLs that are more for internal use. At
least they're definitely not intended to be posted
somewhere for long term useage.
When those things affect a media, it's much better to
reference the media by its id. This can't change, ever.
This is better for races.
Like someone posting a comment while the owner
corrects a typo in the slug.
Now you can have your current directory anywhere and call
runtests.sh with full path and it'll find the way to
nosetests and the tests directory on its own.
Most useful if you're inside the mediagoblin/ dir and can
just call "../runtests.sh".
If you don't give an option argument (starting with a dash)
to runtests.sh then it will append the directory with all
the tests to the commandline of nosetests. That way it will
only search there for tests.
We only ever ran the password changing test here, and not the
half-existing test for changing the bio and homepage.
Fix the tests to actually run both tests. We check that setting bio and
homepage to a valid value works as expected. We also test for invalid
bio (too long) and invalid URLs setting fails. (which is what we want).
- Check that the old /edit/profile/ address redirects to /u/USER/edit/.
- Check that /u/USER/edit/ works as expected
- Check that /u/OTHERUSER/edit/ fails with a 403 Forbidden error when we
are not logged in as that user. The last remaining test that would be
useful is to make a user an admin and make sure the admin can edit
other users' profiles.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Transition from the inconsistent /edit/profile/?username=FOO to
the nicer /u/FOO/edit/. The old pattern will still work and
redirects to the new URL.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Debian testing ships webtest 1.3.4. And it would be nice to
use the base packages.
One of the csrf tests fails on webtest < 1.3.6.
But using a fresh app fixes it.
We have no clue, why exactly.
When we require webtest 1.3.6, change this.
Factor all the migration related stuff out into a new
.db.sql.migration_tools.
First we don't have to load this module for our normal
server.
Second it makes all the import dependencies a little more
cleaner.
We were using lots of tempfiles in the audio and video processing
backends which worked around our workbench system. Still use the
tempfiles package but create them in the workbench directory. This
can help address the uploads of large files (#419) where /tmp might
be a smallish tmpfs and our workbench a real disk.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We copy uploaded media from the queue store to the local workbench
and then to its final destination. The latter was done by simply:
dst.write(src.read()) which is of course evil as it reads the whole
file content into RAM. Which *might* arguably still be OK for
images, but you never know.
Make use of the provided storage() methods that offer chunked copying
rather than opening and fudging with files ourselves.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
These tests, don't need fresh databases, so don't discard and recreate
the tables. This reduces test suite runtime on my laptop from 130 to 96
seconds.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
When the tests want to create a new user, don't fail if it already
exists and just reuse the existing one. This allows us to run tests
without dumping the whole database if that is not needed for the tests.
The upcoming tests for test_edit will make use of this.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
Removing the Mongo InvalidID legacy code removed an explicit check for
"int" for the id lookup. This led the @get_user_media_entry decorator to
fail if we looked up a nonexisting non-numerical slug (it tried to query
the id with a string, which failed). Cast id to int and return 404 in
case it is non-numeric which fixes the regression. It does not fix the
underlying problem of slug_or_id lookups that were discussed.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
We provided a custom GMQuery class that offered a .sort() method for
compatibility with the Mongo syntax. Now that we have settled for sqlalchemy
which uses the order_by() method, we can safely remove this custom class
and move a little closer to "pure" and native sqlalchemy usage.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This concludes the db.sql.* -> db.* move. Our db abstraction layer is
sqlalchemy, so there is no need to a separate db.sql.* hierarchy.
All tests have been run for each of the commit series to make sure
everything works at every step.
Now that sqlalchemy is providing the database abstractions, there is no
need to hide everything in db.sql. sub-modules. It complicates the code
and provides a futher layer of indirection.
Move the db.sql.util.py to db.util.py and adapt the importers.
This is the last remnant that requires us to keep db.sql.fake.py. Use
ModelName.desc() or sqlalchemy.sql.expression.desc(column) to achieve
descending sorts.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
First rename prepare_entry to prepare_queue_task, because
this is really more like what this thing does.
Thanks to Velmont for noting that we do not need a request
in here, but an "app" is good enough. Which means, that
this stuff can be called from tool scripts too.
Previous exif bump copied the newer version into
mediagoblin/tools/extlib/exif but not in extlib/exif. Fix this by
bumping extlib/exif and symlinking to tools/extlib/exif.
Do note that this is still the version fetched from http://sourceforge.net/projects/exif-py/
while the upstream maintainer seems to be active on:
https://github.com/ianare/exif-py
(The sf.net download is quite new though)
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This was one of the last remaining Mongo holdouts and has been removed from
the tree herewith. Good bye, ObjectId.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
