:date: 2016-10-03

=======================
Monday, October 3, 2016
=======================

Failures on Travis
==================

I analyzed some failures on Travis and did lots of commits with
changes to :xfile:`.travis.yml` and :xfile:`requirements.txt` files.

I also started a new documentation page :ref:`lino.ci`.

One error was :message:`/bin/bash: open: command not found` (after
:message:`Writing html report to htmlcov`).  Explanation: when we run
:cmd:`inv cov` on Travis, then we must specify the ``--no-html``
option because :mod:`atelier.invlib`, after creating a html report,
tries to interactively show the produced report. Which fails in a
batch environment.
  
I removed coverage measuring from the :xfile:`.travis.yml` file of
:ref:`xl` because IMO measuring coverage there makes no sense. We do
one coverage in :ref:`book`, and this covers the three projects lino,
xl and book.

I removed usage of `coveralls` because I don't understand why we need
it. (Hamza, don't hesitate to explain me why you added them if there
is a chance that I missed something.)

In the `install` section, I changed :cmd:`pip install .` to :cmd:`pip
install -e .` (i.e. added the `-e` option) because I believe it is not
necessary to to have :command:`pip copy` all the files.

A tough one was :message:`AssertionError: sorry, but this version only
supports 100 named groups`. It received its own ticket :ticket:`1198`
when also :ref:`subha` reported it, he got it when installing a new
docker container.

This was obviously caused by some external change (some new version in
some dependency) because it occurs in all projects which use
weasyprint. Noi is an easy example, there it also failed:
https://travis-ci.org/lino-framework/noi/builds/164577382

16:30 : I can now reproduce it on my machine. And it is mysterious. I
have two virtualenvs, one is my default working environment (called
`py27`), the other one I created it from scratch just for this
problem, it is called `ttt`.  I open two terminal windows (one for
each virtualenv) and in both terminals I got to the directry of
:mod:`lino_noi.projects.team` in my repository (both environments are
using this code in development mode).  I then run the
:xfile:`initdb_tmp` script, and in `py27` it works while in `ttt` it
fails::


    (ttt) luc@doll:~/work/noi/lino_noi/projects/team$ ./initdb_tmp 
    Traceback (most recent call last):
      File "/media/dell1tb/tmp/ttt/bin/django-admin.py", line 5, in <module>
        management.execute_from_command_line()
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
        utility.execute()
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 327, in execute
        django.setup()
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
        apps.populate(settings.INSTALLED_APPS)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
        app_config.import_models(all_models)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
        self.models_module = import_module(models_module_name)
      File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
        __import__(name)
      File "/media/dell1tb/work/lino/lino/modlib/weasyprint/models.py", line 10, in <module>
        from .choicelists import *
      File "/media/dell1tb/work/lino/lino/modlib/weasyprint/choicelists.py", line 25, in <module>
        from weasyprint import HTML
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/weasyprint/__init__.py", line 338, in <module>
        from .css import PARSER, preprocess_stylesheet  # noqa
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/weasyprint/css/__init__.py", line 30, in <module>
        from . import computed_values
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/weasyprint/css/computed_values.py", line 18, in <module>
        from .. import text
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/weasyprint/text.py", line 201, in <module>
        ''')
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/cffi/api.py", line 105, in cdef
        self._cdef(csource, override=override, packed=packed)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/cffi/api.py", line 119, in _cdef
        self._parser.parse(csource, override=override, **options)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/cffi/cparser.py", line 299, in parse
        self._internal_parse(csource)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/cffi/cparser.py", line 304, in _internal_parse
        ast, macros, csource = self._parse(csource)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/cffi/cparser.py", line 260, in _parse
        ast = _get_parser().parse(csource)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/cffi/cparser.py", line 40, in _get_parser
        _parser_cache = pycparser.CParser()
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/pycparser/c_parser.py", line 87, in __init__
        outputdir=taboutputdir)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/pycparser/c_lexer.py", line 66, in build
        self.lexer = lex.lex(object=self, **kwargs)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/pycparser/ply/lex.py", line 911, in lex
        lexobj.readtab(lextab, ldict)
      File "/media/dell1tb/tmp/ttt/local/lib/python2.7/site-packages/pycparser/ply/lex.py", line 233, in readtab
        titem.append((re.compile(pat, lextab._lexreflags | re.VERBOSE), _names_to_funcs(func_name, fdict)))
      File "/media/dell1tb/tmp/ttt/lib/python2.7/re.py", line 194, in compile
        return _compile(pattern, flags)
      File "/media/dell1tb/tmp/ttt/lib/python2.7/re.py", line 249, in _compile
        p = sre_compile.compile(pattern, flags)
      File "/media/dell1tb/tmp/ttt/lib/python2.7/sre_compile.py", line 583, in compile
        "sorry, but this version only supports 100 named groups {}".format(p.pattern.groups)
    AssertionError: sorry, but this version only supports 100 named groups 178
  
Both environments have exactly the same :cmd:`pip freeze`. I manually
updated that of `ttt` one package at a time. Their PYTHONPATH and
LINO_CACHE_ROOT are the same. They are both running the same script in
the same working directory.  And still the problem occurs in one and
does not occur in the other. What am I missing?!

The solution was here:

  https://stackoverflow.com/questions/39829473/cryptography-assertionerror-sorry-but-this-version-only-supports-100-named-gro

So it was a bug in pycparser which occurred only with the *binary*
version. This is why pip freeze was not enough to see the difference!

Anyway the problem in `ttt` disappeared when I did::
  
    $ pip install git+https://github.com/eliben/pycparser@release_v2.14

And so I added the following line to the :xfile:`requirements.txt`
files of concerned projects::
  
    -e git+https://github.com/eliben/pycparser@release_v2.14#egg=pycparser
  
Last steps for supporting Python 3
==================================

We are approaching another historical milestone: our `page on Travis
<https://travis-ci.org/lino-framework>`_ is almost green!

Until now this milestone was waiting for ticket :ticket:`1157` the
most important show stopper.  This :ticket:`1157` is currently waiting
for Gaetans answers to some questions about the general strategy.  But
thanks to Sandro I know that there is a workaround: Stefan Klug has
ported a fork of `appy` to Python 3:
https://pypi.python.org/pypi/appypod/0.9.6 So our milestone, no longer
needs to wait for Gaetan.

Now that the "appy.pod and Python 3" problem is solved, there are --of
course-- quite some some more "trivial" problems in :ref:`xl` and
:ref:`book`.

I started to fix some of them, but then stopped it because this might
be a good task for :ref:`subha`: :ticket:`1199`.