ASCII media type support & fix a bug in file submission error handling

* Added ASCII media processing
* Added ASCII media display
* Added ASCII media type

Rebased from Joar Wandborg's ascii art branch (squashed to remove the
commits borrowing code of dubious license)

Fixed a bug in file submission error handling:
 - Moved file-extension condition out of loop (what did it do there?)
 - Updated file submission tests
 - Changed error handling in file submission, should now report more
   than absolutely necessary.

extlib/inconsolata/INFO.txt

@@ -0,0 +1,4 @@
+Inconsolata
+-----------
+
+This font is found at http://www.levien.com/type/myfonts/inconsolata.html

extlib/inconsolata/OFL_1.1.txt

@@ -0,0 +1,97 @@
+Copyright (c) <dates>, <Copyright Holder> (<URL|email>),
+with Reserved Font Name <Reserved Font Name>.
+Copyright (c) <dates>, <additional Copyright Holder> (<URL|email>),
+with Reserved Font Name <additional Reserved Font Name>.
+Copyright (c) <dates>, <additional Copyright Holder> (<URL|email>).
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded, 
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

mediagoblin/media_types/__init__.py

@@ -69,16 +69,20 @@ def get_media_type_and_manager(filename):
     '''
     Get the media type and manager based on a filename
     '''
-    for media_type, manager in get_media_managers():
     if filename.find('.') > 0:
         # Get the file extension
         ext = os.path.splitext(filename)[1].lower()
     else:
-            raise InvalidFileType(
+        raise Exception(
-                _('Could not find any file extension in "{filename}"').format(
+            _(u'Could not extract any file extension from "{filename}"').format(
                 filename=filename))
 
+    for media_type, manager in get_media_managers():
         # Omit the dot from the extension and match it against
         # the media manager
         if ext[1:] in manager['accepted_extensions']:
             return media_type, manager
+    else:
+        raise FileTypeNotSupported(
+            # TODO: Provide information on which file types are supported
+            _(u'Sorry, I don\'t support that file type :('))

mediagoblin/media_types/ascii/__init__.py

@@ -0,0 +1,27 @@
+# GNU MediaGoblin -- federated, autonomous media hosting
+# Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from mediagoblin.media_types.ascii.processing import process_ascii
+
+
+MEDIA_MANAGER = {
+    "human_readable": "ASCII",
+    "processor": process_ascii, # alternately a string,
+                                # 'mediagoblin.media_types.image.processing'?
+    "display_template": "mediagoblin/media_displays/ascii.html",
+    "default_thumb": "images/media_thumbs/ascii.jpg",
+    "accepted_extensions": [
+        "txt"]}

mediagoblin/media_types/ascii/asciitoimage.py

@@ -0,0 +1,172 @@
+# GNU MediaGoblin -- federated, autonomous media hosting
+# Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import Image
+import ImageFont
+import ImageDraw
+import logging
+import pkg_resources
+import os
+
+_log = logging.getLogger(__name__)
+
+class AsciiToImage(object):
+    '''
+    Converter of ASCII art into image files, preserving whitespace
+
+    kwargs:
+    - font: Path to font file
+      default: fonts/Inconsolata.otf
+    - font_size: Font size, ``int``
+      default: 11
+    '''
+
+    # Font file path
+    _font = None
+
+    _font_size = 11
+
+    # ImageFont instance
+    _if = None
+
+    # ImageFont
+    _if_dims = None
+
+    # Image instance
+    _im = None
+
+    def __init__(self, **kw):
+        if kw.get('font'):
+            self._font = kw.get('font')
+        else:
+            self._font = pkg_resources.resource_filename(
+                'mediagoblin.media_types.ascii',
+                os.path.join('fonts', 'Inconsolata.otf'))
+
+        if kw.get('font_size'):
+            self._font_size = kw.get('font_size')
+
+        _log.info('Setting font to {0}, size {1}'.format(
+                self._font,
+                self._font_size))
+
+        self._if = ImageFont.truetype(
+            self._font,
+            self._font_size)
+
+        #      ,-,-^-'-^'^-^'^-'^-.
+        #     ( I am a wall socket )Oo,  ___
+        #      `-.,.-.,.-.-.,.-.--'     '   `
+        # Get the size, in pixels of the '.' character
+        self._if_dims = self._if.getsize('.')
+        #                               `---'
+
+    def convert(self, text, destination):
+        # TODO: Detect if text is a file-like, if so, act accordingly
+        im = self._create_image(text)
+
+        # PIL's Image.save will handle both file-likes and paths
+        if im.save(destination):
+            _log.info('Saved image in {0}'.format(
+                    destination))
+
+    def _create_image(self, text):
+        '''
+        Write characters to a PIL image canvas.
+
+        TODO:
+        - Character set detection and decoding,
+          http://pypi.python.org/pypi/chardet
+        '''
+        # TODO: Account for alternative line endings
+        lines = text.split('\n')
+
+        line_lengths = [len(i) for i in lines]
+
+        # Calculate destination size based on text input and character size
+        im_dims = (
+            max(line_lengths) * self._if_dims[0],
+            len(line_lengths) * self._if_dims[1])
+
+        _log.info('Destination image dimensions will be {0}'.format(
+                im_dims))
+
+        im = Image.new(
+            'RGBA',
+            im_dims,
+            (255, 255, 255, 0))
+
+        draw = ImageDraw.Draw(im)
+
+        char_pos = [0, 0]
+
+        for line in lines:
+            line_length = len(line)
+
+            _log.debug('Writing line at {0}'.format(char_pos))
+
+            for _pos in range(0, line_length):
+                char = line[_pos]
+
+                px_pos = self._px_pos(char_pos)
+
+                _log.debug('Writing character "{0}" at {1} (px pos {2}'.format(
+                        char,
+                        char_pos,
+                        px_pos))
+
+                draw.text(
+                    px_pos,
+                    char,
+                    font=self._if,
+                    fill=(0, 0, 0, 255))
+
+                char_pos[0] += 1
+
+            # Reset X position, increment Y position
+            char_pos[0] = 0
+            char_pos[1] += 1
+
+        return im
+
+    def _px_pos(self, char_pos):
+        '''
+        Helper function to calculate the pixel position based on
+        character position and character dimensions
+        '''
+        px_pos = [0, 0]
+        for index, val in zip(range(0, len(char_pos)), char_pos):
+                px_pos[index] = char_pos[index] * self._if_dims[index]
+
+        return px_pos
+
+
+if __name__ == "__main__":
+    import urllib
+    txt = urllib.urlopen('file:///home/joar/Dropbox/ascii/install-all-the-dependencies.txt')
+
+    _log.setLevel(logging.DEBUG)
+    logging.basicConfig()
+
+    converter = AsciiToImage()
+
+    converter.convert(txt.read(), '/tmp/test.png')
+
+    '''
+    im, x, y, duration = renderImage(h, 10)
+    print "Rendered image in %.5f seconds" % duration
+    im.save('tldr.png', "PNG")
+    '''

mediagoblin/media_types/ascii/fonts/Inconsolata.otf

@@ -0,0 +1 @@
+../../../../extlib/inconsolata/Inconsolata.otf

mediagoblin/media_types/ascii/processing.py

@@ -0,0 +1,93 @@
+# GNU MediaGoblin -- federated, autonomous media hosting
+# Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+import asciitoimage
+import chardet
+import os
+import Image
+
+from mediagoblin import mg_globals as mgg
+from mediagoblin.processing import create_pub_filepath, THUMB_SIZE
+
+
+def process_ascii(entry):
+    '''
+    Code to process a txt file
+    '''
+    workbench = mgg.workbench_manager.create_workbench()
+    # Conversions subdirectory to avoid collisions
+    conversions_subdir = os.path.join(
+        workbench.dir, 'conversions')
+    os.mkdir(conversions_subdir)
+
+    queued_filepath = entry['queued_media_file']
+    queued_filename = workbench.localized_file(
+        mgg.queue_store, queued_filepath,
+        'source')
+
+    queued_file = file(queued_filename, 'rb')
+
+    with queued_file:
+        queued_file_charset = chardet.detect(queued_file.read())
+
+        queued_file.seek(0)  # Rewind the queued file
+
+        thumb_filepath = create_pub_filepath(
+            entry, 'thumbnail.png')
+
+        tmp_thumb_filename = os.path.join(
+            conversions_subdir, thumb_filepath[-1])
+
+        converter = asciitoimage.AsciiToImage()
+
+        thumb = converter._create_image(
+            queued_file.read())
+
+        with file(tmp_thumb_filename, 'w') as thumb_file:
+            thumb.thumbnail(THUMB_SIZE, Image.ANTIALIAS)
+            thumb.save(thumb_file)
+
+        mgg.public_store.copy_local_to_storage(
+            tmp_thumb_filename, thumb_filepath)
+
+        queued_file.seek(0)
+
+        original_filepath = create_pub_filepath(entry, queued_filepath[-1])
+
+        with mgg.public_store.get_file(original_filepath, 'wb') \
+            as original_file:
+            original_file.write(queued_file.read())
+
+
+        queued_file.seek(0)  # Rewind *again*
+
+        unicode_filepath = create_pub_filepath(entry, 'unicode.txt')
+
+        with mgg.public_store.get_file(unicode_filepath, 'wb') \
+                as unicode_file:
+            unicode_file.write(
+                    unicode(queued_file.read().decode(
+                        queued_file_charset['encoding'])).encode(
+                    'ascii',
+                    'xmlcharrefreplace'))
+
+    mgg.queue_store.delete_file(queued_filepath)
+    entry['queued_media_file'] = []
+    media_files_dict = entry.setdefault('media_files', {})
+    media_files_dict['thumb'] = thumb_filepath
+    media_files_dict['unicode'] = unicode_filepath
+    media_files_dict['original'] = original_filepath
+
+    entry.save()

mediagoblin/static/css/base.css

@@ -402,3 +402,15 @@ table.media_panel th {
   margin-top: 10px;
   margin-left: 10px;
 }
+
+/* ASCII art */
+
+@font-face {
+  font-family: Inconsolata;
+  src: local('Inconsolata'), url('../fonts/Inconsolata.otf') format('opentype')
+}
+
+.ascii-wrapper pre {
+  font-family: Inconsolata, monospace;
+  line-height: 1em;
+}

mediagoblin/static/fonts/Inconsolata.otf

@@ -0,0 +1 @@
+../../../extlib/inconsolata/Inconsolata.otf

mediagoblin/submit/views.py

@@ -128,9 +128,13 @@ def submit_start(request):
 
                 return redirect(request, "mediagoblin.user_pages.user_home",
                                 user=request.user.username)
-            except InvalidFileType, exc:
+            except Exception as e:
+                '''
+                This section is intended to catch exceptions raised in 
+                mediagobling.media_types
+                '''
                 submit_form.file.errors.append(
-                    _(u'Invalid file type.'))
+                    e)
 
     return render_to_response(
         request,

mediagoblin/templates/mediagoblin/media_displays/ascii.html

@@ -0,0 +1,40 @@
+{#
+# GNU MediaGoblin -- federated, autonomous media hosting
+# Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#}
+
+{% extends 'mediagoblin/user_pages/media.html' %}
+
+{% block mediagoblin_media %}
+  <div class="ascii-wrapper">
+    <pre>
+      {%- autoescape False -%}
+      {{- request.app.public_store.get_file(
+             media['media_files']['unicode']).read()|string -}}
+      {%- endautoescape -%}
+    </pre>
+  </div>
+  {% if 'original' in media.media_files %}
+  <p>
+    <a href="{{ request.app.public_store.file_url(
+	     media['media_files']['original']) }}">
+      {%- trans -%}
+        Original
+      {%- endtrans -%}
+    </a>
+  </p>
+  {% endif %}    
+{% endblock %}

mediagoblin/templates/mediagoblin/media_displays/image.html

@@ -1 +1,19 @@
+{#
+# GNU MediaGoblin -- federated, autonomous media hosting
+# Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#}
+
 {% extends 'mediagoblin/user_pages/media.html' %}

mediagoblin/templates/mediagoblin/media_displays/video.html

@@ -1,3 +1,21 @@
+{#
+# GNU MediaGoblin -- federated, autonomous media hosting
+# Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#}
+
 {% extends 'mediagoblin/user_pages/media.html' %}
 
 {% block mediagoblin_media %}

mediagoblin/tests/test_submission.py

@@ -1,3 +1,4 @@
+
 # GNU MediaGoblin -- federated, autonomous media hosting
 # Copyright (C) 2011 MediaGoblin contributors.  See AUTHORS.
 #
@@ -16,6 +17,7 @@
 
 import urlparse
 import pkg_resources
+import re
 
 from nose.tools import assert_equal, assert_true, assert_false
 
@@ -216,7 +218,8 @@ class TestSubmission:
 
         context = template.TEMPLATE_TEST_CONTEXT['mediagoblin/submit/start.html']
         form = context['submit_form']
-        assert form.file.errors == [u'Invalid file type.']
+        assert re.match(r'^Could not extract any file extension from ".*?"$', str(form.file.errors[0]))
+        assert len(form.file.errors) == 1
 
         # NOTE: The following 2 tests will ultimately fail, but they
         #   *will* pass the initial form submission step.  Instead,