heckyel/mediagoblin
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Tune up the celery and production deployments documentation.

Browse Source
This commit is contained in:
Ben Sturmfels 2021-03-29 16:39:46 +11:00
parent a14f4f93eb
commit eb2aae91da
2 changed files with 84 additions and 141 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

80
docs/source/siteadmin/deploying.rst
View File

@ -498,10 +498,10 @@ Type ``Ctrl-c`` to exit the above server test and ``exit`` or
Run MediaGoblin as a system service Run MediaGoblin as a system service
----------------------------------- -----------------------------------
To ensure MediaGoblin is automatically started and restarted in case of To ensure MediaGoblin is automatically started and restarted in case
problems, we need to run it as a system service. If your operating system uses of problems, we need to run it as system services. If your operating
Systemd, you can use Systemd ``service files`` to manage both the Celery and system uses Systemd, you can use Systemd ``service files`` to manage
Paste processes. both the Celery and Paste processes as described below.
In the Systemd configuration below, MediaGoblin log files are kept in In the Systemd configuration below, MediaGoblin log files are kept in
the ``/var/log/mediagoblin`` directory. Create the directory and give the ``/var/log/mediagoblin`` directory. Create the directory and give
@ -512,9 +512,31 @@ it the proper permissions::
Place the following service files in the ``/etc/systemd/system/`` Place the following service files in the ``/etc/systemd/system/``
directory. The first file should be named directory. The first file should be named
``mediagoblin-celeryd.service``. Be sure to modify it to suit your ``mediagoblin-paster.service``. Be sure to modify it to suit your
environment's setup: environment's setup:
.. code-block:: bash
# Set the WorkingDirectory and Environment values to match your environment.
[Unit]
Description=Mediagoblin
[Service]
Type=simple
User=mediagoblin
Group=mediagoblin
Environment=CELERY_ALWAYS_EAGER=false
WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin
ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
/srv/mediagoblin.example.org/mediagoblin/paste.ini \
--log-file=/var/log/mediagoblin/mediagoblin.log \
--server-name=main
[Install]
WantedBy=multi-user.target
The second file should be named ``mediagoblin-celeryd.service``:
.. code-block:: bash .. code-block:: bash
# Set the WorkingDirectory and Environment values to match your environment. # Set the WorkingDirectory and Environment values to match your environment.
@ -536,40 +558,19 @@ environment's setup:
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target
For details on this approach with a separate Celery process, see
The second file should be named ``mediagoblin-paster.service``: :ref:`background-media-processing`.
.. code-block:: bash
# Set the WorkingDirectory and Environment values to match your environment.
[Unit]
Description=Mediagoblin
[Service]
Type=simple
User=mediagoblin
Group=mediagoblin
Environment=CELERY_ALWAYS_EAGER=false
WorkingDirectory=/srv/mediagoblin.example.org/mediagoblin
ExecStart=/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
/srv/mediagoblin.example.org/mediagoblin/paste.ini \
--log-file=/var/log/mediagoblin/mediagoblin.log \
--server-name=main
[Install]
WantedBy=multi-user.target
Enable these processes to start at boot by entering:: Enable these processes to start at boot by entering::
sudo systemctl enable mediagoblin-celeryd.service && sudo systemctl enable mediagoblin-paster.service sudo systemctl enable mediagoblin-paster.service
sudo systemctl enable mediagoblin-celeryd.service
Start the processes for the current session with:: Start the processes for the current session with::
sudo systemctl start mediagoblin-celeryd.service
sudo systemctl start mediagoblin-paster.service sudo systemctl start mediagoblin-paster.service
sudo systemctl start mediagoblin-celeryd.service
If either command above gives you an error, you can investigate the cause of If either command above gives you an error, you can investigate the cause of
the error by entering either of:: the error by entering either of::
@ -577,15 +578,8 @@ the error by entering either of::
sudo systemctl status mediagoblin-celeryd.service sudo systemctl status mediagoblin-celeryd.service
sudo systemctl status mediagoblin-paster.service sudo systemctl status mediagoblin-paster.service
The above ``systemctl status`` command is also useful if you ever want to The above ``systemctl status`` command is also useful if you ever want to
confirm that a process is still running. If you make any changes to the ".service" confirm that a process is still running.
files, you can reload the service files and restart MediaGoblin by entering::
sudo systemctl daemon-reload
sudo systemctl restart mediagoblin-celeryd.service
sudo systemctl restart mediagoblin-paster.service
Assuming the above was successful, you should now have a MediaGoblin Assuming the above was successful, you should now have a MediaGoblin
server that will continue to operate, even after being restarted. server that will continue to operate, even after being restarted.
@ -602,6 +596,14 @@ To restart MediaGoblin after making configuration changes, run::
sudo systemctl restart mediagoblin-celeryd.service sudo systemctl restart mediagoblin-celeryd.service
sudo systemctl restart mediagoblin-paster.service sudo systemctl restart mediagoblin-paster.service
If you make any changes to the ".service" files, you must first issue
a `daemon-reload` command to refresh Systemd and then restart
MediaGoblin with::
sudo systemctl daemon-reload
sudo systemctl restart mediagoblin-celeryd.service
sudo systemctl restart mediagoblin-paster.service
What next? What next?
---------- ----------

143
docs/source/siteadmin/production-deployments.rst
View File

@ -11,13 +11,12 @@
Dedication along with this software. If not, see Dedication along with this software. If not, see
<http://creativecommons.org/publicdomain/zero/1.0/>. <http://creativecommons.org/publicdomain/zero/1.0/>.
========================================= =================================================
Considerations for Production Deployments Further Considerations for Production Deployments
========================================= =================================================
This document contains a number of suggestions for deploying This page extends upon our ":doc:`deploying`" guide to describe some common
MediaGoblin in actual production environments. Consider issues affecting production deployments.
":doc:`deploying`" for a basic overview of how to deploy MediaGoblin.
Should I Keep Open Registration Enabled? Should I Keep Open Registration Enabled?
@ -43,122 +42,64 @@ We hope to have a better solution to this situation shortly. We
apologize for the inconvenience in the meanwhile. apologize for the inconvenience in the meanwhile.
Security Considerations Confidential Files
----------------------- ------------------
.. warning:: .. warning::
The directory ``user_dev/crypto/`` contains some very The directory ``user_dev/crypto/`` contains confidential information. In
sensitive files. particular, the ``itsdangeroussecret.bin`` is important for the security of
Especially the ``itsdangeroussecret.bin`` is very important login sessions. Make sure not to publish its contents anywhere. If the
for session security. Make sure not to leak its contents anywhere. contents gets leaked nevertheless, delete your file and restart the server,
If the contents gets leaked nevertheless, delete your file so that it creates a new secret key. All previous login sessions will be
and restart the server, so that it creates a new secret key. invalidated.
All previous sessions will be invalidated.
.. _init-script: .. _background-media-processing:
Alternative init scripts Background Media Processing
------------------------ ---------------------------
If your system does not use Systemd, you can use the following command as the ":doc:`deploying`" covers use of a separate Celery process, but this sections
basis for an init script: describes this in more detail.
.. code-block:: bash
CELERY_ALWAYS_EAGER=true \
/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
/srv/mediagoblin.example.org/mediagoblin/paste.ini \
--pid-file=/var/run/mediagoblin.pid \
--server-name=main
The above configuration places MediaGoblin in "always eager" mode
with Celery, this means that submissions of content will be processed
synchronously, and the user will advance to the next page only after
processing is complete. If we take Celery out of "always eager mode,"
the user will be able to immediately return to the MediaGoblin site
while processing is ongoing. In these cases, use the following command
as the basis for your script:
.. code-block:: bash
CELERY_ALWAYS_EAGER=false \
/srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
/srv/mediagoblin.example.org/mediagoblin/paste.ini \
--pid-file=/var/run/mediagoblin.pid \
--server-name=main
Members of the MediaGoblin community have provided init scripts for the
following GNU/Linux distributions:
Arch Linux
* `MediaGoblin - ArchLinux rc.d scripts
<http://whird.jpope.org/2012/04/14/mediagoblin-archlinux-rcd-scripts>`_
by `Jeremy Pope <http://jpope.org/>`_
* `Mediagoblin init script on Archlinux
<http://chimo.chromic.org/2012/03/01/mediagoblin-init-script-on-archlinux/>`_
by `Chimo <http://chimo.chromic.org/>`_
You can reference these scripts to create an init script for your own operating
system. Similar scripts will be in your system's ``/etc/init.d/``
or ``/etc/rc.d/`` directory, but the specifics of an init script will vary from
one distribution to the next.
.. _separate-celery:
Separate celery
---------------
":doc:`deploying`" describes a configuration with a separate Celery process, but
the following section covers this in more detail.
MediaGoblin uses `Celery`_ to handle heavy and long-running tasks. Celery can MediaGoblin uses `Celery`_ to handle heavy and long-running tasks. Celery can
be launched in two ways: be launched in two ways:
1. Embedded in the MediaGoblin WSGI application [#f-mediagoblin-wsgi-app]_. 1. **Embedded in the main MediaGoblin web application.** This is the way
This is the way ``./lazyserver.sh`` does it for you. It's simple as you ``./lazyserver.sh`` does it for you. It's simple as you only have to run one
only have to run one process. The only bad thing with this is that the process. The only bad thing with this is that the heavy and long-running
heavy and long-running tasks will run *in* the webserver, keeping the user tasks will run *in* the webserver, keeping the user waiting each time some
waiting each time some heavy lifting is needed as in for example processing heavy lifting is needed as in for example processing a video. This could lead
a video. This could lead to problems as an aborted connection will halt any to problems as an aborted connection will halt any processing and since most
processing and since most front-end web servers *will* terminate your front-end web servers *will* terminate your connection if it doesn't get any
connection if it doesn't get any response from the MediaGoblin WSGI response from the MediaGoblin web application in a while. This approach is
application in a while. suitable for development, small sites or when primarily using :doc:`command
line uploads <commandline-upload>`.
2. As a separate process communicating with the MediaGoblin WSGI application 2. **As a separate web application and media processing application
via a `broker`_. This offloads the heavy lifting from the MediaGoblin WSGI (recommended).** In this approach, the MediaGoblin web application delegates
all media processing to a task queue via a `broker`_ (task queue). This is
the approach used in our :doc:`deployment guide <deploying>`, with RabbitMQ
as the broker. This offloads the heavy lifting from the MediaGoblin web
application and users will be able to continue to browse the site while the application and users will be able to continue to browse the site while the
media is being processed in the background. media is being processed in the background. This approach provided the best
user experience and is recommended for production sites.
The choice between these two behaviours is controlled by the
``CELERY_ALWAYS_EAGER`` environment variable. Specifying ``true`` instructs
MediaGoblin to processing media within the web application while you wait.
Specifying ``false`` instructs MediaGoblin to use background processing.
.. _`broker`: http://docs.celeryproject.org/en/latest/getting-started/brokers/ .. _`broker`: http://docs.celeryproject.org/en/latest/getting-started/brokers/
.. _`celery`: http://www.celeryproject.org/ .. _`celery`: http://www.celeryproject.org/
.. [#f-mediagoblin-wsgi-app] The MediaGoblin WSGI application is the part that
of MediaGoblin that processes HTTP requests.
To launch Celery separately from the MediaGoblin WSGI application:
1. Make sure that the ``CELERY_ALWAYS_EAGER`` environment variable is unset or
set to ``false`` when launching the MediaGoblin WSGI application.
2. Start the ``celeryd`` main process with
.. code-block:: bash
CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd
If you use our example Systemd ``service files``, Celery will be set to the
"CELERY_ALWAYS_EAGER=false" value by default. This will provide your users
with the best user experience, as all media processing will be done in the
background.
.. _sentry: .. _sentry:
Set up sentry to monitor exceptions Error Monitoring with Sentry
----------------------------------- ----------------------------
We have a plugin for `raven`_ integration, see the ":doc:`/plugindocs/raven`" We have a plugin for `raven`_ integration, see the ":doc:`/plugindocs/raven`"
documentation. documentation.