heckyel/mediagoblin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Moves hacking howto and design decisions docs to wiki

Browse Source
This commit is contained in:
Will Kahn-Greene 2011-07-31 23:07:13 -04:00
parent f9372f6c8b
commit 65e7ce634c
7 changed files with 7 additions and 683 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/codebase.rst
View File

@ -21,7 +21,7 @@ various recipes for getting things done.
for where we hang out.
For more information on how to get started hacking on GNU MediaGoblin,
see :ref:`hacking-howto`.
see `the wiki <http://wiki.mediagoblin.org/>`_.
Software Stack

6
docs/source/contributinghowto.rst
View File

@ -46,8 +46,8 @@ Here are some things you can do today:
**Write/Fix some code**
If you are a coder and you're looking to code, check out the
:ref:`hacking-howto`. We even have tips on *becoming* a coder
and we're willing to help you!
`wiki <http://wiki.mediagoblin.org/`_. We even have tips on
*becoming* a coder and we're willing to help you!
**Send encouragement**
@ -135,7 +135,7 @@ the project, Chris Webber, will make a custom drawing of a goblin
dedicated specifically to you.
For why we're doing copyright assignment, see the
:ref:`design-decisions-chapter`.
`wiki <http://wiki.mediagoblin.org/>`_.
.. _filing-bugs:

4
docs/source/deploymenthowto.rst
View File

@ -12,5 +12,5 @@ Step 3: Write the deployment guide and profit!
But seriously, this is a stub since we're not quite there (yet) but if
you want to see where we are now, you can try to run the latest
development version by following the instructions at
:ref:`hacking-howto`.
development version by following the instructions on
`the wiki <http://wiki.mediagoblin.org/>`_.

329
docs/source/designdecisions.rst
View File

@ -1,329 +0,0 @@
.. _design-decisions-chapter:
==================
Design Decisions
==================
.. contents:: Sections
:local:
This chapter talks a bit about design decisions.
Why GNU MediaGoblin?
====================
Chris and Will on "Why GNU MediaGoblin":
Chris came up with the name MediaGoblin. The name is pretty fun.
It merges the idea that this is a Media hosting project with
Goblin which sort of sounds like gobbling. Here's a piece of
software that gobbles up your media for all to see.
`According to Wikipedia <http://en.wikipedia.org/wiki/Goblin>`_, a
goblin is:
a legendary evil or mischievous illiterate creature, described
as grotesquely evil or evil-like phantom
So are we evil? No. Are we mischievous or illiterate? Not
really. So what kind of goblin are we thinking about? We're
thinking about these goblins:
.. figure:: goblin.png
:alt: Cute goblin with a beret.
*Figure 1: Cute goblin with a beret. llustrated by Chris
Webber*
.. figure:: snugglygoblin.png
:scale: 50%
:alt: Snuggly goblin with a beret.
*Figure 2: Snuggly goblin. Illustrated by Karen Rustad*
Those are pretty cute goblins. Those are the kinds of goblins
we're thinking about.
Chris started doing work on the project after thinking about it
for a year. Then, after talking with Matt and Rob, it became an
official GNU project. Thus we now call it GNU MediaGoblin.
That's a lot of letters, though, so in the interest of brevity and
facilitating easier casual conversation and balancing that with
what's important to us, we have the following rules:
1. "GNU MediaGoblin" is the name we're going to use in all official
capacities: web site, documentation, press releases, ...
2. In casual conversation, it's ok to use more casual names.
3. If you're writing about the project, we ask that you call it GNU
MediaGoblin.
4. If you don't like the name, we kindly ask you to take a deep
breath, think a happy thought about cute little goblins playing
on a playground and taking cute pictures of themselves, and let
it go. (Will added this one.)
Why Python
==========
Chris Webber on "Why Python":
Because I know Python, love Python, am capable of actually making
this thing happen in Python (I've worked on a lot of large free
software web applications before in Python, including `Miro
Community`_, the `Miro Guide`_, a large portion of `Creative
Commons`_, and a whole bunch of things while working at `Imaginary
Landscape`_). Me starting a project like this makes sense if it's
done in Python.
You might say that PHP is way more deployable, that Rails has way
more cool developers riding around on fixie bikes---and all of
those things are true. But I know Python, like Python, and think
that Python is pretty great. I do think that deployment in Python
is not as good as with PHP, but I think the days of shared hosting
are (thankfully) coming to an end, and will probably be replaced
by cheap virtual machines spun up on the fly for people who want
that sort of stuff, and Python will be a huge part of that future,
maybe even more than PHP will. The deployment tools are getting
better. Maybe we can use something like Silver Lining. Maybe we
can just distribute as ``.debs`` or ``.rpms``. We'll figure it
out when we get there.
Regardless, if I'm starting this project, which I am, it's gonna
be in Python.
.. _Miro Community: http://mirocommunity.org/
.. _Miro Guide: http://miroguide.org/
.. _Creative Commons: http://creativecommons.org/
.. _Imaginary Landscape: http://www.imagescape.com/
Why WSGI Minimalism
===================
Chris Webber on "Why WSGI Minimalism":
If you notice in the technology list I list a lot of components
that are very "django-like", but not actually `Django`_
components. What can I say, I really like a lot of the ideas in
Django! Which leads to the question: why not just use Django?
While I really like Django's ideas and a lot of its components, I
also feel that most of the best ideas in Django I want have been
implemented as good or even better outside of Django. I could
just use Django and replace the templating system with Jinja2, and
the form system with wtforms, and the database with MongoDB and
MongoKit, but at that point, how much of Django is really left?
I also am sometimes saddened and irritated by how coupled all of
Django's components are. Loosely coupled yes, but still coupled.
WSGI has done a good job of providing a base layer for running
applications on and if you know how to do it yourself [1]_, it's
not hard or many lines of code at all to bind them together
without any framework at all (not even say `Pylons`_, `Pyramid`_
or `Flask`_ which I think are still great projects, especially for
people who want this sort of thing but have no idea how to get
started). And even at this already really early stage of writing
MediaGoblin, that glue work is mostly done.
Not to say I don't think Django isn't great for a lot of things.
For a lot of stuff, it's still the best, but not for MediaGoblin,
I think.
One thing that Django does super well though is documentation. It
still has some faults, but even with those considered I can hardly
think of any other project in Python that has as nice of
documentation as Django. It may be worth learning some lessons on
documentation from Django [2]_, on that note.
I'd really like to have a good, thorough hacking-howto and
deployment-howto, especially in the former making some notes on
how to make it easier for Django hackers to get started.
.. _Django: http://www.djangoproject.com/
.. _Pylons: http://pylonshq.com/
.. _Pyramid: http://docs.pylonsproject.org/projects/pyramid/dev/
.. _Flask: http://flask.pocoo.org/
.. [1] http://pythonpaste.org/webob/do-it-yourself.html
.. [2] http://pycon.blip.tv/file/4881071/
Why MongoDB
===========
Chris Webber on "Why MongoDB":
In case you were wondering, I am not a NOSQL fanboy, I do not go
around telling people that MongoDB is web scale. Actually my
choice for MongoDB isn't scalability, though scaling up really
nicely is a pretty good feature and sets us up well in case large
volume sites eventually do use MediaGoblin. But there's another
side of scalability, and that's scaling down, which is important
for federation, maybe even more important than scaling up in an
ideal universe where everyone ran servers out of their own
housing. As a memory-mapped database, MongoDB is pretty hungry,
so actually I spent a lot of time debating whether the inability
to scale down as nicely as something like SQL has with sqlite
meant that it was out.
But I decided in the end that I really want MongoDB, not for
scalability, but for flexibility. Schema evolution pains in SQL
are almost enough reason for me to want MongoDB, but not quite.
The real reason is because I want the ability to eventually handle
multiple media types through MediaGoblin, and also allow for
plugins, without the rigidity of tables making that difficult. In
other words, something like::
{"title": "Me talking until you are bored",
"description": "blah blah blah",
"media_type": "audio",
"media_data": {
"length": "2:30",
"codec": "OGG Vorbis"},
"plugin_data": {
"licensing": {
"license": "http://creativecommons.org/licenses/by-sa/3.0/"}}}
Being able to just dump media-specific information in a media_data
hashtable is pretty great, and even better is having a plugin
system where you can just let plugins have their own entire
key-value space cleanly inside the document that doesn't interfere
with anyone else's stuff. If we were to let plugins to deposit
their own information inside the database, either we'd let plugins
create their own tables which makes SQL migrations even harder
than they already are, or we'd probably end up creating a table
with a column for key, a column for value, and a column for type
in one huge table called "plugin_data" or something similar. (Yo
dawg, I heard you liked plugins, so I put a database in your
database so you can query while you query.) Gross.
I also don't want things to be too loose so that we forget or lose
the structure of things, and that's one reason why I want to use
MongoKit, because we can cleanly define a much structure as we
want and verify that documents match that structure generally
without adding too much bloat or overhead (MongoKit is a pretty
lightweight wrapper and doesn't inject extra MongoKit-specific
stuff into the database, which is nice and nicer than many other
ORMs in that way).
Why Sphinx for documentation
============================
Will Kahn-Greene on "Why Sphinx":
`Sphinx`_ is a fantastic tool for organizing documentation for a
Python-based project that makes it pretty easy to write docs that
are readable in source form and can be "compiled" into HTML, LaTeX
and other formats.
There are other doc systems out there, but given that GNU
MediaGoblin is being written in Python and I've done a ton of
documentation using Sphinx, it makes sense to use Sphinx for now.
.. _Sphinx: http://sphinx.pocoo.org/
Why AGPLv3 and CC0?
===================
Chris, Brett, Will, Rob, Matt, et al curated into a story where
everyone is the hero by Will on "Why AGPLv3 and CC0":
The `AGPL v3`_ preserves the freedoms guaranteed by the GPL v3 in
the context of software as a service. Using this license ensures
that users of the service have the ability to examine the source,
deploy their own instance, and implement their own version. This
is really important to us and a core mission component of this
project. Thus we decided that the software parts should be under
this license.
However, the project is made up of more than just software:
there's CSS, images, and other output-related things. We wanted
the templates/images/css side of the project all permissive and
permissive in the same absolutely permissive way. We're waiving
our copyrights to non-software things under the CC0 waiver.
That brings us to the templates where there's some code and some
output. The template engine we're using is called Jinja2. It
mixes HTML markup with Python code to render the output of the
software. We decided the templates are part of the output of the
software and not the software itself. We wanted the output of the
software to be licensed in a hassle-free way so that when someone
deploys their own GNU MediaGoblin instance with their own
templates, they don't have to deal with the copyleft aspects of
the AGPLv3 and we'd be fine with that because the changes they're
making are identity-related. So at first we decided to waive our
copyrights to the templates with a CC0 waiver and then add an
exception to the AGPLv3 for the software such that the templates
can make calls into the software and yet be a separately licensed
work. However, Brett brought up the question of whether this
allows some unscrupulous person to make changes to the software
through the templates in such a way that they're not bound by the
AGPLv3: i.e. a loophole. We thought about this loophole and
between this and the extra legalese involved in the exception to
the AGPLv3, we decided that it's just way simpler if the templates
were also licensed under the AGPLv3.
Then we have the licensing for the documentation. Given that the
documentation is tied to the software content-wise, we don't feel
like we have to worry about ensuring freedom of the documentation
or worry about attribution concerns. Thus we're waiving our
copyrights to the documentation under CC0 as well.
Lastly, we have branding. This covers logos and other things that
are distinctive to GNU MediaGoblin that we feel represents this
project. Since we don't currently have any branding, this is an
open issue, but we're thinking we'll go with a CC BY-SA license.
By licensing in this way, we make sure that users of the software
receive the freedoms that the AGPLv3 ensures regardless of what
fate befalls this project.
So to summarize:
* software (Python, JavaScript, HTML templates): licensed
under AGPLv3
* non-software things (CSS, images, video): copyrights waived
under CC0 because this is output of the software
* documentation: copyrights waived under CC0 because it's not part
of the software
* branding assets: we're kicking this can down the road, but
probably CC BY-SA
This is all codified in the ``COPYING`` file.
.. _AGPL v3: http://www.gnu.org/licenses/agpl.html
.. _CC0 v1: http://creativecommons.org/publicdomain/zero/1.0/
Why (non-mandatory) copyright assignment?
=========================================
Chris Webber on "Why copyright assignment?":
GNU MediaGoblin is a GNU project with non-mandatory but heavily
encouraged copyright assignment to the FSF. Most, if not all, of
the core contributors to GNU MediaGoblin will have done a
copyright assignment, but unlike some other GNU projects, it isn't
required here. We think this is the best choice for GNU
MediaGoblin: it ensures that the Free Software Foundation may
protect the software by enforcing the AGPL if the FSF sees fit,
but it also means that we can immediately merge in changes from a
new contributor. It also means that some significant non-FSF
contributors might also be able to enforce the AGPL if seen fit.
Again, assignment is not mandatory, but it is heavily encouraged,
even incentivized: significant contributors who do a copyright
assignment to the FSF are eligible to have a unique goblin drawing
produced for them by the project's main founder, Christopher Allan
Webber. See :ref:`contributing-howto-chapter` for details.

2
docs/source/git.rst
View File

@ -221,4 +221,4 @@ because he doesn't need it anymore.
How to learn git
================
Check out :ref:`hacking-howto-git`!
Check out `the wiki <http://wiki.mediagoblin.org/>`_.

345
docs/source/hackinghowto.rst
View File

@ -1,345 +0,0 @@
.. _hacking-howto:
===============
Hacking HOWTO
===============
.. contents:: Sections
:local:
So you want to hack on GNU MediaGoblin?
=======================================
First thing to do is check out the `Web site
<http://mediagoblin.org/join/>`_ where we list all the project
infrastructure including:
* the IRC channel
* the mailing list
* the issue tracker
Additionally, we have information on how to get involved, who to talk
to, what needs to be worked on, and other things besides!
Second thing to do is take a look at :ref:`codebase-chapter` where
we've started documenting how GNU MediaGoblin is built and how to add
new things.
Third you'll need to :ref:`get the requirements
<get-requirements-section>`.
Fourth, you'll need to build a development environment. We use buildout,
but if you want to use virtualenv, there's a set of mediocre not-very-supported
steps in the `wiki <https://gitorious.org/mediagoblin/pages/Home>`_.
.. _get-requirements-section:
Getting requirements
====================
First, you need to have the following installed before you can build
an environment for hacking on GNU MediaGoblin:
* Python 2.6 or 2.7 - http://www.python.org/
You'll need Python as well as the dev files for building modules.
* python-lxml - http://lxml.de/
* git - http://git-scm.com/
* MongoDB - http://www.mongodb.org/
If you're running Debian GNU/Linux or a Debian-derived distribution
such as Mint or Ubuntu, running the following should install these
requirements::
sudo apt-get install mongodb git-core python python-dev \
python-lxml
On Fedora::
yum install mongodb-server python-paste-deploy python-paste-script \
git-core python python-devel python-lxml
.. YouCanHelp::
If you have instructions for other GNU/Linux distributions to set
up requirements, let us know!
.. _hacking-with-buildout:
How to set up and maintain an environment for hacking with buildout
===================================================================
**Requirements**
No additional requirements.
**Create a development environment**
After installing the requirements, follow these steps:
1. Clone the repository::
git clone git://gitorious.org/mediagoblin/mediagoblin.git
2. Bootstrap and run buildout::
cd mediagoblin
python bootstrap.py && ./bin/buildout
That's it! Using this method, buildout should create a ``user_dev``
directory, in which certain things will be stored (media, beaker
session stuff, etc). You can change this, but for development
purposes this default should be fine.
**Updating for dependency changes**
While hacking on GNU MediaGoblin over time, you'll eventually have to
update your development environment because the dependencies have
changed. To do that, run::
./bin/buildout && ./bin/gmg migrate
**Updating for code changes**
You don't need to do anything---code changes are automatically
available.
**Deleting your buildout**
At some point, you may want to delete your buildout. Perhaps it's to
start over. Perhaps it's to test building development environments
with buildout.
To do this, do::
rm -rf bin develop-eggs eggs mediagoblin.egg-info parts user_dev
Running the server
==================
If you want to get things running quickly and without hassle, just
run::
./lazyserver.sh
This will start up a python server where you can begin playing with
mediagoblin. It will also run celery in "always eager" mode so you
don't have to start a separate process for it.
This is fine in development, but if you want to actually run celery
separately for testing (or deployment purposes), you'll want to run
the server independently::
./bin/paster serve paste.ini --reload
Running celeryd
===============
If you aren't using ./lazyserver.sh or otherwise aren't running celery
in always eager mode, you'll need to do this if you want your media to
process and actually show up. It's probably a good idea in
development to have the web server (above) running in one terminal and
celeryd in another window.
Run::
CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd
Running the test suite
======================
Run::
./runtests.sh
Running a shell
===============
If you want a shell with your database pre-setup and an instantiated
application ready and at your fingertips....
Run::
./bin/gmg shell
Troubleshooting
===============
pymongo.errors.AutoReconnect: could not find master/primary
-----------------------------------------------------------
If you see this::
pymongo.errors.AutoReconnect: could not find master/primary
then make sure mongodb is installed and running.
If it's installed, check the mongodb log. On my machine, that's
``/var/log/mongodb/mongodb.log``. If you see something like::
old lock file: /var/lib/mongodb/mongod.lock. probably means...
in that case you might have had an unclean shutdown. Try::
sudo mongod --repair
If that didn't work, just delete the lock file and relaunch mongodb.
Anyway, then start the mongodb server in whatever way is appropriate
for your distro / OS.
pkg_resources.DistributionNotFound: distribute
----------------------------------------------
If you get this while running buildout::
pkg_resources.DistributionNotFound: distribute
Try this commmand instead::
python bootstrap.py --distribute && ./bin/buildout
Wiping your user data
=====================
.. Note::
Unless you're doing development and working on and testing creating
a new instance, you will probably never have to do this. Will
plans to do this work and thus he documented it.
.. YouCanHelp::
If you're familiar with MongoDB, we'd love to get a `script that
removes all the GNU MediaGoblin data from an existing instance
<http://bugs.foocorp.net/issues/296>`_. Let us know!
Quickstart for Django programmers
=================================
We're not using Django, but the codebase is very Django-like in its
structure.
* ``routing.py`` is like ``urls.py`` in Django
* ``models.py`` has mongokit ORM definitions
* ``views.py`` is where the views go
We're using MongoDB. Basically, instead of a relational database with
tables, you have a big JSON structure which acts a lot like a Python
dict.
.. YouCanHelp::
If there are other things that you think would help orient someone
new to GNU MediaGoblin but coming from Django, let us know!
Bite-sized bugs to start with
=============================
**May 3rd, 2011**: We don't have a list of bite-sized bugs, yet, but
this is important to us. If you're interested in things to work on,
let us know on `the mailing list <http://mediagoblin.org/join/>`_ or
on the `IRC channel <http://mediagoblin.org/join/>`_.
Tips for people new to coding
=============================
Learning Python
---------------
GNU MediaGoblin is written using a programming language called `Python
<http://python.org/>`_.
There are two different incompatible iterations of Python which I'll
refer to as Python 2 and Python 3. GNU MediaGoblin is written in
Python 2 and requires Python 2.6 or 2.7. At some point, we might
switch to Python 3, but that's a future thing.
You can learn how to code in Python 2 from several excellent books
that are freely available on the Internet:
* `Learn Python the Hard Way <http://learnpythonthehardway.org/>`_
* `Dive Into Pyton <http://diveintopython.org/>`_
* `Python for Software Design <http://www.greenteapress.com/thinkpython/>`_
* `A Byte of Python <http://www.swaroopch.com/notes/Python>`_
These are all excellent texts.
.. YouCanHelp::
If you know of other good quality Python tutorials and Python
tutorial videos, let us know!
Learning Libraries GNU MediaGoblin uses
---------------------------------------
GNU MediaGoblin uses a variety of libraries in order to do what it
does. These libraries are listed in the :ref:`codebase-chapter`
along with links to the project Web sites and documentation for the
libraries.
There are a variety of Python-related conferences every year that have
sessions covering many aspects of these libraries. You can find them
at `Python Miro Community <http://python.mirocommunity.org>`_ [0]_.
.. [0] This is a shameless plug. Will Kahn-Greene runs Python Miro
Community.
If you have questions or need help, find us on the mailing list and on
IRC.
.. _hacking-howto-git:
Learning git
------------
git is an interesting and very powerful tool. Like all powerful
tools, it has a learning curve.
If you're new to git, we highly recommend the following resources for
getting the hang of it:
* `Learn Git <http://learn.github.com/p/intro.html>`_ --- the GitHub
intro to git
* `Pro Git <http://progit.org/book/>`_ --- fantastic book
* `Git casts <http://gitcasts.com/>`_ --- screencast covering git
usage
* `Git Reference <http://gitref.org/>`_ --- Git reference that makes
it easier to get the hang of git if you're coming from other version
control systems
There's also a git mission at `OpenHatch <http://openhatch.org/>`_.
Learning other utilities
------------------------
The `OpenHatch <http://openhatch.org/>`_ site has a series of
`training missions <http://openhatch.org/missions/>`_ which are
designed to help you learn how to use these tools.
If you're new to tar, diff, patch and git, we highly recommend you sign
up with OpenHatch and do the missions.

2
docs/source/index.rst
View File

@ -15,11 +15,9 @@ Table of Contents:
mediagoblin
contributinghowto
deploymenthowto
hackinghowto
theminghowto
git
codebase
designdecisions
vision