Bug #25252
le nombre de connexions concurrentes pour les tests vaut exactement le max toléré par postgres par défaut
Statut:
Fermé
Priorité:
Normal
Assigné à:
-
Version cible:
-
Début:
12 juillet 2018
Echéance:
% réalisé:
0%
Temps estimé:
Patch proposed:
Oui
Planning:
Description
Et donc, test_lookup_user_transaction échoue, de temps en temps, pour l'environnement virtuel tox Python3/Django1.11 avec backend postgres :
(publik-env) paul@amok:~/src/django-mellon$ tox -e py3-dj111-pg
GLOB sdist-make: /home/paul/src/django-mellon/setup.py
py3-dj111-pg inst-nodeps: /tmp/tox-paul/django-mellon/dist/django-mellon-1.2.37.2.ga13e45e.zip
py3-dj111-pg installed: atomicwrites==1.1.5,attrs==18.1.0,beautifulsoup4==4.6.0,certifi==2018.4.16,chardet==3.0.4,coverage==4.5.1,cssselect==1.0.3,Django==1.11.14,django-mellon===1.2.37.2.ga13e45e,django-webtest==1.9.2,httmock==1.2.6,idna==2.7,isodate==0.6.0,lxml==4.2.3,mock==2.0.0,more-itertools==4.2.0,pbr==4.1.0,pkg-resources==0.0.0,pluggy==0.6.0,psycopg2==2.7.5,py==1.5.4,pyquery==1.4.0,pytest==3.6.3,pytest-cov==2.5.1,pytest-django==3.3.2,pytest-mock==1.10.0,pytest-random==0.2,pytz==2018.5,requests==2.19.1,six==1.11.0,urllib3==1.23,waitress==1.1.0,WebOb==1.8.2,WebTest==2.0.30
py3-dj111-pg runtests: PYTHONHASHSEED='2878064241'
py3-dj111-pg runtests: commands[0] | ./getlasso3.sh
'/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/lasso.py' -> '/usr/lib/python3/dist-packages/lasso.py'
'/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/_lasso.cpython-36m-x86_64-linux-gnu.so' -> '/usr/lib/python3/dist-packages/_lasso.cpython-36m-x86_64-linux-gnu.so'
'/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/_lasso.cpython-37m-x86_64-linux-gnu.so' -> '/usr/lib/python3/dist-packages/_lasso.cpython-37m-x86_64-linux-gnu.so'
py3-dj111-pg runtests: commands[1] | py.test --random tests
===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
Tests are shuffled using seed number 392039017015.
Django settings: testsettings (from environment variable)
rootdir: /home/paul/src/django-mellon, inifile:
plugins: random-0.2, mock-1.10.0, django-3.3.2, cov-2.5.1, django-webtest-1.9.2
collected 43 items
tests/test_utils.py . [ 2%]
tests/test_views.py . [ 4%]
tests/test_utils.py . [ 6%]
tests/test_default_adapter.py . [ 9%]
tests/test_utils.py ... [ 16%]
tests/test_views.py . [ 18%]
tests/test_sso_slo.py .. [ 23%]
tests/test_utils.py . [ 25%]
tests/test_views.py .. [ 30%]
tests/test_default_adapter.py .. [ 34%]
tests/test_sso_slo.py . [ 37%]
tests/test_default_adapter.py . [ 39%]
tests/test_sso_slo.py . [ 41%]
tests/test_utils.py .. [ 46%]
tests/test_default_adapter.py . [ 48%]
tests/test_utils.py .... [ 58%]
tests/test_default_adapter.py . [ 60%]
tests/test_views.py . [ 62%]
tests/test_default_adapter.py . [ 65%]
tests/test_federation_utils.py . [ 67%]
tests/test_utils.py . [ 69%]
tests/test_views.py ... [ 76%]
tests/test_default_adapter.py F. [ 81%]
tests/test_utils.py . [ 83%]
tests/test_views.py . [ 86%]
tests/test_sso_slo.py . [ 88%]
tests/test_views.py ... [ 95%]
tests/test_sso_slo.py .. [100%]
========================================================================================== FAILURES ===========================================================================================
________________________________________________________________________________ test_lookup_user_transaction _________________________________________________________________________________
self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f053c2f19b0>
def ensure_connection(self):
"""
Guarantees that a connection to the database is established.
"""
if self.connection is None:
with self.wrap_database_errors:
> self.connect()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:213:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f053c2f19b0>
def connect(self):
"""Connects to the database. Assumes that the connection is closed."""
# Check for invalid configurations.
self.check_settings()
# In case the previous connection was closed while in an atomic block
self.in_atomic_block = False
self.savepoint_ids = []
self.needs_rollback = False
# Reset parameters defining when to close the connection
max_age = self.settings_dict['CONN_MAX_AGE']
self.close_at = None if max_age is None else time.time() + max_age
self.closed_in_transaction = False
self.errors_occurred = False
# Establish the connection
conn_params = self.get_connection_params()
> self.connection = self.get_new_connection(conn_params)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:189:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f053c2f19b0>, conn_params = {'database': 'mellon-test-postgresql_psycopg2'}
def get_new_connection(self, conn_params):
> connection = Database.connect(**conn_params)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/postgresql/base.py:176:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dsn = 'dbname=mellon-test-postgresql_psycopg2', connection_factory = None, cursor_factory = None, kwargs = {'database': 'mellon-test-postgresql_psycopg2'}, kwasync = {}
def connect(dsn=None, connection_factory=None, cursor_factory=None, **kwargs):
"""
Create a new database connection.
The connection parameters can be specified as a string:
conn = psycopg2.connect("dbname=test user=postgres password=secret")
or using a set of keyword arguments:
conn = psycopg2.connect(database="test", user="postgres", password="secret")
Or as a mix of both. The basic connection parameters are:
- *dbname*: the database name
- *database*: the database name (only as keyword argument)
- *user*: user name used to authenticate
- *password*: password used to authenticate
- *host*: database host address (defaults to UNIX socket if not provided)
- *port*: connection port number (defaults to 5432 if not provided)
Using the *connection_factory* parameter a different class or connections
factory can be specified. It should be a callable object taking a dsn
argument.
Using the *cursor_factory* parameter, a new default cursor factory will be
used by cursor().
Using *async*=True an asynchronous connection will be created. *async_* is
a valid alias (for Python versions where ``async`` is a keyword).
Any other keyword parameter will be passed to the underlying client
library: the list of supported parameters depends on the library version.
"""
kwasync = {}
if 'async' in kwargs:
kwasync['async'] = kwargs.pop('async')
if 'async_' in kwargs:
kwasync['async_'] = kwargs.pop('async_')
if dsn is None and not kwargs:
raise TypeError('missing dsn and no parameters')
dsn = _ext.make_dsn(dsn, **kwargs)
> conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E psycopg2.OperationalError: FATAL: les emplacements de connexions restants sont réservés pour les connexions
E superutilisateur non relatif à la réplication
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/psycopg2/__init__.py:130: OperationalError
The above exception was the direct cause of the following exception:
transactional_db = None, concurrency = 100
def test_lookup_user_transaction(transactional_db, concurrency):
adapter = DefaultAdapter()
p = ThreadPool(concurrency)
def f(i):
# sqlite has a default lock timeout of 5s seconds between different access to the same in
# memory DB
if connection.vendor == 'sqlite':
connection.cursor().execute('PRAGMA busy_timeout = 400000')
try:
return adapter.lookup_user(idp, saml_attributes)
finally:
connection.close()
> users = p.map(f, range(concurrency))
tests/test_default_adapter.py:71:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.6/multiprocessing/pool.py:266: in map
return self._map_async(func, iterable, mapstar, chunksize).get()
/usr/lib/python3.6/multiprocessing/pool.py:644: in get
raise self._value
/usr/lib/python3.6/multiprocessing/pool.py:119: in worker
result = (True, func(*args, **kwds))
/usr/lib/python3.6/multiprocessing/pool.py:44: in mapstar
return list(map(*args))
tests/test_default_adapter.py:68: in f
return adapter.lookup_user(idp, saml_attributes)
mellon/adapters.py:201: in lookup_user
saml_identifiers__issuer=issuer)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/models/manager.py:85: in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/models/query.py:374: in get
num = len(clone)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/models/query.py:232: in __len__
self._fetch_all()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/models/query.py:1118: in _fetch_all
self._result_cache = list(self._iterable_class(self))
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/models/query.py:53: in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/models/sql/compiler.py:887: in execute_sql
cursor = self.connection.cursor()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:254: in cursor
return self._cursor()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:229: in _cursor
self.ensure_connection()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:213: in ensure_connection
self.connect()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/utils.py:94: in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/utils/six.py:685: in reraise
raise value.with_traceback(tb)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:213: in ensure_connection
self.connect()
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/base/base.py:189: in connect
self.connection = self.get_new_connection(conn_params)
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/django/db/backends/postgresql/base.py:176: in get_new_connection
connection = Database.connect(**conn_params)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dsn = 'dbname=mellon-test-postgresql_psycopg2', connection_factory = None, cursor_factory = None, kwargs = {'database': 'mellon-test-postgresql_psycopg2'}, kwasync = {}
def connect(dsn=None, connection_factory=None, cursor_factory=None, **kwargs):
"""
Create a new database connection.
The connection parameters can be specified as a string:
conn = psycopg2.connect("dbname=test user=postgres password=secret")
or using a set of keyword arguments:
conn = psycopg2.connect(database="test", user="postgres", password="secret")
Or as a mix of both. The basic connection parameters are:
- *dbname*: the database name
- *database*: the database name (only as keyword argument)
- *user*: user name used to authenticate
- *password*: password used to authenticate
- *host*: database host address (defaults to UNIX socket if not provided)
- *port*: connection port number (defaults to 5432 if not provided)
Using the *connection_factory* parameter a different class or connections
factory can be specified. It should be a callable object taking a dsn
argument.
Using the *cursor_factory* parameter, a new default cursor factory will be
used by cursor().
Using *async*=True an asynchronous connection will be created. *async_* is
a valid alias (for Python versions where ``async`` is a keyword).
Any other keyword parameter will be passed to the underlying client
library: the list of supported parameters depends on the library version.
"""
kwasync = {}
if 'async' in kwargs:
kwasync['async'] = kwargs.pop('async')
if 'async_' in kwargs:
kwasync['async_'] = kwargs.pop('async_')
if dsn is None and not kwargs:
raise TypeError('missing dsn and no parameters')
dsn = _ext.make_dsn(dsn, **kwargs)
> conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E django.db.utils.OperationalError: FATAL: les emplacements de connexions restants sont réservés pour les connexions
E superutilisateur non relatif à la réplication
/tmp/tox-paul/django-mellon/py3-dj111-pg/lib/python3.6/site-packages/psycopg2/__init__.py:130: OperationalError
============================================================================= 1 failed, 42 passed in 3.03 seconds =============================================================================
ERROR: InvocationError: '/tmp/tox-paul/django-mellon/py3-dj111-pg/bin/py.test --random tests'
___________________________________________________________________________________________ summary ___________________________________________________________________________________________
ERROR: py3-dj111-pg: commands failed
Est-ce qu'on aurait pas intérêt à réduire ce nombre de connexions à quelque chose un peu en dessous du max pris en charge par défaut ?
Fichiers
Révisions associées
Historique
Mis à jour par Paul Marillonnet il y a presque 6 ans
- Fichier 0001-tests-reduce-number-of-concurrent-connections-below-.patch 0001-tests-reduce-number-of-concurrent-connections-below-.patch ajouté
- Statut changé de Nouveau à Solution proposée
- Patch proposed changé de Non à Oui
Mis à jour par Benjamin Dauvergne il y a presque 6 ans
Non, c'est des tests volontairement exagérés, il vaut mieux augmenter le nombre de connexion du postgres sur jenkins. Coté tests on peut vraisemblablement mettre un test qui skip le test si on détecte que le nombre de connexions autorisés est inférieur à un certain seuile (je pense que ça fait partie des settings lisibles).
Mis à jour par Paul Marillonnet il y a presque 6 ans
- Fichier 0001-skip-test-if-number-of-concurrent-connections-above-.patch 0001-skip-test-if-number-of-concurrent-connections-above-.patch ajouté
Un patch qui passe simplement le test en question.
Mis à jour par Paul Marillonnet il y a presque 6 ans
- Fichier 0001-skip-test-if-number-of-concurrent-connections-above-.patch 0001-skip-test-if-number-of-concurrent-connections-above-.patch ajouté
Avec la fermeture du curseur c'est mieux :)
Mis à jour par Benjamin Dauvergne il y a presque 6 ans
- Statut changé de Solution proposée à Solution validée
Mais plus idiomatique:
with connection.cursor() as c: ....
pas besoin de close dans ce cas.
Mis à jour par Paul Marillonnet il y a presque 6 ans
Merci. Il me manque #24531 pour que les tests passent avec la dernière version de lasso.
Mis à jour par Paul Marillonnet il y a presque 6 ans
Poussé jeudi dernier, oublié de mettre à jour le ticket
commit 349a4ec3bc700680d24d108d6dc241f04632669e
Author: Paul Marillonnet <pmarillonnet@entrouvert.com>
Date: Thu Jul 12 16:00:23 2018 +0200
skip test if number of concurrent connections above default pg limit (#25252)
Mis à jour par Frédéric Péters il y a plus de 5 ans
- Statut changé de Solution validée à Résolu (à déployer)
Mis à jour par Benjamin Dauvergne il y a plus de 5 ans
- Statut changé de Résolu (à déployer) à Fermé
skip test if number of concurrent connections above default pg limit (#25252)