Project

General

Profile

Development #14630

Avoir un import-template générique

Added by Benjamin Dauvergne over 2 years ago. Updated 3 months ago.

Status:
Solution déployée
Priority:
Normal
Category:
-
Start date:
17 Jan 2017
Due date:
% Done:

0%

Patch proposed:
Yes
Planning:
No

Description

Mon idée ce serait dans chaque application d'avoir des commandes import/export-site qui prennent un fichier ou affiche le résultat en sortie standard.

Tout ce que ferait import-template c'est savoir qu'il faut aller chercher le fichier dans /var/lib/<application>/skeletons/<template-name> mais pour le reste il ferait appel à call_command('import-site', ...) qui sera le standard pour nos applications (c'est déja import_site dans combo, à voir si je standardise sur le underscore dans passerelle aussi).

Par souci d'uniformité je proposerai bien d'avoir la même chose dans w.c.s. à un moment.

0001-agent-add-generic-import_template-management-command.patch View (23.3 KB) Christophe Siraut, 12 Feb 2019 05:38 PM


Related issues

Related to Passerelle - Development #13887: possibilité d'export/import de paramétrage Fermé 07 Nov 2016
Related to Combo - Development #15665: Étendre import/export _site comme les autres applis Publik Solution déployée 29 Mar 2017
Related to Publik - Development #20770: Pouvoir déployer Publik depuis une UI Nouveau 18 Dec 2017
Related to Authentic 2 - Development #16514: commande de gestion pour exporter/importer des rôles et OU Fermé 26 May 2017
Related to Hobo - Development #29762: ordonnancement des import-template Nouveau 15 Jan 2019
Related to Hobo - Bug #32469: mauvais chemin par défaut lors de l'import des templates Solution déployée 18 Apr 2019

Associated revisions

Revision 03a2aa8e (diff)
Added by Christophe Siraut 3 months ago

agent: add generic import_template management command (#14630)

History

#1 Updated by Benjamin Dauvergne over 2 years ago

#2 Updated by Benjamin Dauvergne over 2 years ago

L'implémentation par défaut devrait appeler la commande suivante si elle existe:

import_site --if-empty /var/lib/<application>/skeletons/<template_name>

#3 Updated by Frédéric Péters about 2 years ago

  • Related to Development #15665: Étendre import/export _site comme les autres applis Publik added

#4 Updated by Frédéric Péters over 1 year ago

#5 Updated by Frédéric Péters about 1 year ago

  • Related to Development #16514: commande de gestion pour exporter/importer des rôles et OU added

#8 Updated by Benjamin Dauvergne 6 months ago

L'idée c'est d'ajouter à hobo une commande import-template, qui prend en paramètre template_name; le code pour l'appeler est déjà là, voir #14154, et attends tranquillement qu'on implémente ces commandes, et ce que je propose c'est que par défaut cette commande lance un import_site --if-empty /var/lib/<app_name>/skeletons/<template_name>.json si ce fichier existe.

À nous de déployer via paquets ou autre ces templates ensuite (on a encore jamais discuter ce ça mais faut déjà qu'on arrive à un moment où ça marche sur la plupart des applications); actuellement passerelle, combo, chrono et authentic ont un commande import_site.

#9 Updated by Christophe Siraut 6 months ago

  • Assignee set to Christophe Siraut

#10 Updated by Christophe Siraut 5 months ago

Je n'arrive pas à grand chose, (je suis aveuglément le descriptif); si on veut que ça avance est-ce que tu prendrais en charge ce ticket Benjamin? J'ai tenté hobo/agent/common/management/commands/import_template.py :

class Command(BaseCommand):

    def add_arguments(self, parser):
        parser.add_argument('template_name', type=str)

    def handle(self, *args, **kwargs):

        APP_NAME = ?
        template = '/var/lib/%s/skeletons/%s.json' % (APP_NAME, kwargs['template_name'])

        if 'import_site' in get_commands():

            if not os.path.isfile(template):
                raise(Exeception('Template not found'))

            call_command('import_site', '--if-empty', template) 

#11 Updated by Benjamin Dauvergne 5 months ago

Christophe Siraut a écrit :

Je n'arrive pas à grand chose, (je suis aveuglément le descriptif); si on veut que ça avance est-ce que tu prendrais en charge ce ticket Benjamin? J'ai tenté hobo/agent/common/management/commands/import_template.py :

[...]

Et ?

#12 Updated by Christophe Siraut 5 months ago

Et ?

je ne sais pas comment trouver APP_NAME depuis une commande de management.

#13 Updated by Benjamin Dauvergne 5 months ago

settings.PROJECT_NAME.

#14 Updated by Christophe Siraut 5 months ago

  • Patch proposed changed from No to Yes
  • File 0001-agent-add-generic-import_template-management-command.patch added
  • Status changed from Nouveau to Solution proposée

une version de base.

#15 Updated by Benjamin Dauvergne 5 months ago

Y a pu qu'à faire des test pour voir ce que ça donne en vrai, par exemple dans les tests authentic et passerelle intégrés.

#16 Updated by Christophe Siraut 5 months ago

  • File 0001-agent-add-generic-import_template-management-command.patch added
  • Status changed from Solution proposée to En cours

En passant, avec un test passerelle naïf je reçois :

  chris@devinst:~/src/hobo$ DEBIAN_CONFIG_COMMON=debian/debian_config_common.py PASSERELLE_SETTINGS_FILE=tests_passerelle/settings.py DJANGO_SETTINGS_MODULE=passerelle.settings python -m pytest tests_passerelle/test_import_template.py 
  […]
  ../passerelle/passerelle/base/management/commands/import_site.py:28: in handle
  E   AttributeError: 'list' object has no attribute 'copy'

#17 Updated by Frédéric Péters 5 months ago

E AttributeError: 'list' object has no attribute 'copy'

Je ne sais de quelle commande tu as construit :

jstr = '''[{
  "model": "common.role",
  …

mais un export de passerelle ne ressemble pas à ça. (ça doit être un dictionnaire avec "apiusers" et "resources" comme clés.)

#18 Updated by Benjamin Dauvergne 5 months ago

#19 Updated by Christophe Siraut 5 months ago

Je ne sais de quelle commande tu as construit […]

dumpdata :/

#20 Updated by Christophe Siraut 5 months ago

  • File 0001-agent-add-generic-import_template-management-command.patch added
  • Status changed from En cours to Solution proposée

Avec un test passerelle.

#21 Updated by Benjamin Dauvergne 5 months ago

Push the branch dude.

#22 Updated by Christophe Siraut 4 months ago

  • File deleted (0001-agent-add-generic-import_template-management-command.patch)

#23 Updated by Christophe Siraut 4 months ago

  • File deleted (0001-agent-add-generic-import_template-management-command.patch)

#24 Updated by Christophe Siraut 4 months ago

  • File deleted (0001-agent-add-generic-import_template-management-command.patch)

#25 Updated by Christophe Siraut 4 months ago

J'ai simplifié le patch et les tests.

Les tests coté passerelle passent (avec un json minimaliste); ceux de authentic échouent (avec un json complet issu d'un export). Pour commencer le dictionnaire renvoyé par authentic2.data_transfer.export_site contient des entrées unicode tandis que celles de passerelle sont toutes en bytes. J'ai utilisé byteify() pour contourner le problème coté a2 mais je ne me rend pas compte s'il y a lieu de corriger quelque chose en amont.

Ensuite voici les différences observées entre authentic2.data_transfer.export_site() et les données présentes dans le template json. Il y a, outre le problème d'encodage, des uuid qui diffèrent, mais je ne me rend pas compte si le json que je lui donne à manger est représentatif de ce qu'on veut faire (ou si les rôles de bases devraient être exclus du usecase)

(publik-env) chris@devinst:~/src/hobo$ DEBIAN_CONFIG_COMMON=debian/debian_config_common.py DJANGO_SETTINGS_MODULE=authentic2.settings AUTHENTIC2_SETTINGS_FILE=tests_authentic/settings.py py.test -k test_import_template tests_authentic -vv -s
/home/chris/src/authentic/src/authentic2/settings.py:167: UserWarning: journald will not be used directly, please install python-systemd

================================================================================================ test session starts =================================================================================================
platform linux2 -- Python 2.7.13, pytest-4.1.1, py-1.7.0, pluggy-0.8.1 -- /home/chris/envs/publik-env/bin/python2
cachedir: .pytest_cache
Django settings: authentic2.settings (from environment variable)
rootdir: /home/chris/src/hobo, inifile:
plugins: mock-1.10.0, django-3.4.5, django-webtest-1.9.2
collected 7 items / 6 deselected                                                                                                                                                                                     

tests_authentic/test_hobo_deploy.py::test_import_template Creating test database for alias 'default' ('test_authentic2_multitenant')...
=== Running migrate for schema public
Operations to perform:
  Synchronize unmigrated apps: auth2_auth, authentic2_auth_saml, authentic2_idp_saml, authentic2_provisionning_ldap, disco_service, django_select2, django_tables2, gadjo, jquery, jquery_ui, manager, messages, multitenant, rest_framework, select2, staticfiles
  Apply all migrations: a2_rbac, admin, attribute_aggregator, auth, auth2_ssl, authentic2, authentic2_auth_fc, authentic2_auth_oidc, authentic2_idp_cas, authentic2_idp_oidc, common, contenttypes, custom_user, django_rbac, idp, mellon, nonce, saml, sessions
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying django_rbac.0001_initial... OK
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying custom_user.0001_initial... OK
  Applying auth.0002_auto_20150323_1720... OK
  Applying authentic2.0001_initial... OK
  Applying authentic2.0002_auto_20150320_1418... OK
  Applying authentic2.0003_auto_20150409_1840... OK
  Applying authentic2.0004_service... OK
  Applying a2_rbac.0001_initial... OK
  Applying a2_rbac.0002_role_external_id... OK
  Applying a2_rbac.0003_partial_unique_index_on_name_and_slug... OK
  Applying a2_rbac.0004_auto_20150523_0028... OK
  Applying a2_rbac.0005_auto_20150526_1406... OK
  Applying a2_rbac.0006_auto_20150619_1056... OK
  Applying a2_rbac.0007_auto_20150708_1337... OK
  Applying a2_rbac.0008_auto_20150810_1953... OK
  Applying a2_rbac.0009_partial_unique_index_on_permission... OK
  Applying a2_rbac.0010_auto_20160209_1417... OK
  Applying a2_rbac.0011_auto_20160209_1511... OK
  Applying a2_rbac.0013_auto_20170629_0007... OK
  Applying a2_rbac.0014_auto_20170711_1024... OK
  Applying a2_rbac.0015_organizationalunit_validate_emails... OK
  Applying a2_rbac.0016_auto_20171208_1429... OK
  Applying a2_rbac.0017_organizationalunit_user_can_reset_password... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying attribute_aggregator.0001_initial... OK
  Applying idp.0001_initial... OK
  Applying saml.0001_initial... OK
  Applying saml.0002_ease_federation_migration... OK
  Applying saml.0002_auto_20150320_1245... OK
  Applying saml.0003_merge... OK
  Applying saml.0004_auto_20150410_1438... OK
  Applying saml.0005_make_liberty_provider_inherit_from_service... OK
  Applying saml.0006_restore_foreign_keys... OK
  Applying saml.0007_copy_service_ptr_id_to_old_id... OK
  Applying saml.0008_alter_foreign_keys... OK
  Applying saml.0009_auto... OK
  Applying saml.0010_auto... OK
  Applying saml.0011_auto... OK
  Applying saml.0012_auto_20150526_2239... OK
  Applying saml.0013_auto_20150617_1004... OK
  Applying saml.0014_auto_20150617_1216... OK
  Applying saml.0015_auto_20150915_2032... OK
  Applying saml.0016_auto_20150915_2041... OK
  Applying idp.0002_auto_20150526_2239... OK
  Applying idp.0003_auto_20150915_2041... OK
  Applying attribute_aggregator.0002_auto_20150409_1840... OK
  Applying attribute_aggregator.0003_auto_20150526_2239... OK
  Applying attribute_aggregator.0004_auto_20150915_2041... OK
  Applying authentic2_idp_cas.0001_initial... OK
  Applying auth2_ssl.0001_initial... OK
  Applying custom_user.0002_auto_20150410_1823... OK
  Applying auth.0003_auto_20150410_1657... OK
  Applying auth.0004_user... OK
  Applying auth.0005_auto_20150526_2303... OK
  Applying auth2_ssl.0002_auto_20150409_1840... OK
  Applying custom_user.0003_auto_20150504_1410... OK
  Applying custom_user.0004_user_ou... OK
  Applying custom_user.0005_auto_20150522_1527... OK
  Applying custom_user.0006_auto_20150527_1212... OK
  Applying custom_user.0007_auto_20150610_1527... OK
  Applying custom_user.0008_auto_20150617_1606... OK
  Applying custom_user.0009_auto_20150810_1953... OK
  Applying custom_user.0010_auto_20160307_1418... OK
  Applying authentic2.0005_service_ou... OK
  Applying authentic2.0006_conditional_slug_index... OK
  Applying authentic2.0007_auto_20150523_0028... OK
  Applying authentic2.0008_auto_20160204_1415... OK
  Applying authentic2.0009_auto_20160211_2247... OK
  Applying authentic2.0010_attributevalue_multiple... OK
  Applying authentic2.0011_auto_20160211_2253... OK
  Applying authentic2.0012_auto_20160211_2255... OK
  Applying authentic2.0013_auto_20160211_2258... OK
  Applying authentic2.0014_attributevalue_verified... OK
  Applying authentic2.0015_auto_20160621_1711... OK
  Applying custom_user.0011_manual_attribute_values_for_name_fields... OK
  Applying custom_user.0012_user_modified... OK
  Applying custom_user.0013_user_email_verified... OK
  Applying custom_user.0014_set_email_verified... OK
  Applying custom_user.0015_auto_20170707_1653... OK
  Applying custom_user.0016_auto_20180925_1107... OK
  Applying authentic2.0016_attribute_disabled... OK
  Applying authentic2.0017_modify_attribute_serialization... OK
  Applying authentic2.0018_auto_20170524_0842... OK
  Applying authentic2.0019_auto_20170309_1529... OK
  Applying authentic2.0020_delete_federatedid... OK
  Applying authentic2.0021_attribute_order... OK
  Applying authentic2.0022_attribute_scopes... OK
  Applying authentic2.0023_auto_20181031_0900... OK
  Applying authentic2_auth_fc.0001_initial... OK
  Applying authentic2_auth_oidc.0001_initial... OK
  Applying authentic2_auth_oidc.0002_oidcprovider_token_revocation_endpoint... OK
  Applying authentic2_auth_oidc.0003_oidcprovider_show... OK
  Applying authentic2_auth_oidc.0004_auto_20171017_1522... OK
  Applying authentic2_auth_oidc.0005_oidcprovider_slug... OK
  Applying authentic2_auth_oidc.0006_oidcprovider_claims_parameter_supported... OK
  Applying authentic2_idp_cas.0002_auto_20150410_1438... OK
  Applying authentic2_idp_cas.0003_auto_20150415_2223... OK
  Applying authentic2_idp_cas.0004_create_services... OK
  Applying authentic2_idp_cas.0005_alter_field_service_ptr... OK
  Applying authentic2_idp_cas.0006_copy_proxy_m2m... OK
  Applying authentic2_idp_cas.0007_alter_service... OK
  Applying authentic2_idp_cas.0008_alter_foreign_keys... OK
  Applying authentic2_idp_cas.0009_alter_related_models... OK
  Applying authentic2_idp_cas.0010_copy_service_ptr_id_to_old_id... OK
  Applying authentic2_idp_cas.0011_remove_old_id_restore_proxy... OK
  Applying authentic2_idp_cas.0012_copy_service_proxy_to_m2m... OK
  Applying authentic2_idp_cas.0013_delete_model_service_proxy2... OK
  Applying authentic2_idp_cas.0014_auto_20151204_1606... OK
  Applying authentic2_idp_cas.0015_auto_20170406_1825... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying authentic2_idp_oidc.0001_initial... OK
  Applying authentic2_idp_oidc.0002_auto_20170121_2346... OK
  Applying authentic2_idp_oidc.0003_auto_20170329_1259... OK
  Applying authentic2_idp_oidc.0004_auto_20170324_1426... OK
  Applying authentic2_idp_oidc.0005_authorization_mode... OK
  Applying authentic2_idp_oidc.0006_auto_20170720_1054... OK
  Applying authentic2_idp_oidc.0007_oidcclient_has_api_access... OK
  Applying authentic2_idp_oidc.0008_oidcclient_idtoken_duration... OK
  Applying authentic2_idp_oidc.0009_auto_20180313_1156... OK
  Applying authentic2_idp_oidc.0010_oidcclaim... OK
  Applying authentic2_idp_oidc.0011_auto_20180808_1546... OK
  Applying common.0001_initial... OK
  Applying common.0002_auto_20160105_1702... OK
  Applying django_rbac.0002_organizationalunit_permission_role_roleparenting... OK
  Applying django_rbac.0003_add_max_aggregate_for_postgres... OK
  Applying django_rbac.0004_auto_20150708_1337... OK
  Applying django_rbac.0005_auto_20171209_1106... OK
  Applying mellon.0001_initial... OK
  Applying nonce.0001_initial... OK
  Applying saml.0017_auto_20170710_1738... OK
  Applying sessions.0001_initial... OK

=== Creating schema authentic_example_net
=== Running migrate for schema authentic_example_net
Operations to perform:
  Apply all migrations: a2_rbac, admin, attribute_aggregator, auth, auth2_ssl, authentic2, authentic2_auth_fc, authentic2_auth_oidc, authentic2_idp_cas, authentic2_idp_oidc, common, contenttypes, custom_user, django_rbac, idp, mellon, nonce, saml, sessions
Running migrations:
  Applying django_rbac.0001_initial... OK
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying custom_user.0001_initial... OK
  Applying auth.0002_auto_20150323_1720... OK
  Applying authentic2.0001_initial... OK
  Applying authentic2.0002_auto_20150320_1418... OK
  Applying authentic2.0003_auto_20150409_1840... OK
  Applying authentic2.0004_service... OK
  Applying a2_rbac.0001_initial... OK
  Applying a2_rbac.0002_role_external_id... OK
  Applying a2_rbac.0003_partial_unique_index_on_name_and_slug... OK
  Applying a2_rbac.0004_auto_20150523_0028... OK
  Applying a2_rbac.0005_auto_20150526_1406... OK
  Applying a2_rbac.0006_auto_20150619_1056... OK
  Applying a2_rbac.0007_auto_20150708_1337... OK
  Applying a2_rbac.0008_auto_20150810_1953... OK
  Applying a2_rbac.0009_partial_unique_index_on_permission... OK
  Applying a2_rbac.0010_auto_20160209_1417... OK
  Applying a2_rbac.0011_auto_20160209_1511... OK
  Applying a2_rbac.0013_auto_20170629_0007... OK
  Applying a2_rbac.0014_auto_20170711_1024... OK
  Applying a2_rbac.0015_organizationalunit_validate_emails... OK
  Applying a2_rbac.0016_auto_20171208_1429... OK
  Applying a2_rbac.0017_organizationalunit_user_can_reset_password... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying attribute_aggregator.0001_initial... OK
  Applying idp.0001_initial... OK
  Applying saml.0001_initial... OK
  Applying saml.0002_ease_federation_migration... OK
  Applying saml.0002_auto_20150320_1245... OK
  Applying saml.0003_merge... OK
  Applying saml.0004_auto_20150410_1438... OK
  Applying saml.0005_make_liberty_provider_inherit_from_service... OK
  Applying saml.0006_restore_foreign_keys... OK
  Applying saml.0007_copy_service_ptr_id_to_old_id... OK
  Applying saml.0008_alter_foreign_keys... OK
  Applying saml.0009_auto... OK
  Applying saml.0010_auto... OK
  Applying saml.0011_auto... OK
  Applying saml.0012_auto_20150526_2239... OK
  Applying saml.0013_auto_20150617_1004... OK
  Applying saml.0014_auto_20150617_1216... OK
  Applying saml.0015_auto_20150915_2032... OK
  Applying saml.0016_auto_20150915_2041... OK
  Applying idp.0002_auto_20150526_2239... OK
  Applying idp.0003_auto_20150915_2041... OK
  Applying attribute_aggregator.0002_auto_20150409_1840... OK
  Applying attribute_aggregator.0003_auto_20150526_2239... OK
  Applying attribute_aggregator.0004_auto_20150915_2041... OK
  Applying authentic2_idp_cas.0001_initial... OK
  Applying auth2_ssl.0001_initial... OK
  Applying custom_user.0002_auto_20150410_1823... OK
  Applying auth.0003_auto_20150410_1657... OK
  Applying auth.0004_user... OK
  Applying auth.0005_auto_20150526_2303... OK
  Applying auth2_ssl.0002_auto_20150409_1840... OK
  Applying custom_user.0003_auto_20150504_1410... OK
  Applying custom_user.0004_user_ou... OK
  Applying custom_user.0005_auto_20150522_1527... OK
  Applying custom_user.0006_auto_20150527_1212... OK
  Applying custom_user.0007_auto_20150610_1527... OK
  Applying custom_user.0008_auto_20150617_1606... OK
  Applying custom_user.0009_auto_20150810_1953... OK
  Applying custom_user.0010_auto_20160307_1418... OK
  Applying authentic2.0005_service_ou... OK
  Applying authentic2.0006_conditional_slug_index... OK
  Applying authentic2.0007_auto_20150523_0028... OK
  Applying authentic2.0008_auto_20160204_1415... OK
  Applying authentic2.0009_auto_20160211_2247... OK
  Applying authentic2.0010_attributevalue_multiple... OK
  Applying authentic2.0011_auto_20160211_2253... OK
  Applying authentic2.0012_auto_20160211_2255... OK
  Applying authentic2.0013_auto_20160211_2258... OK
  Applying authentic2.0014_attributevalue_verified... OK
  Applying authentic2.0015_auto_20160621_1711... OK
  Applying custom_user.0011_manual_attribute_values_for_name_fields... OK
  Applying custom_user.0012_user_modified... OK
  Applying custom_user.0013_user_email_verified... OK
  Applying custom_user.0014_set_email_verified... OK
  Applying custom_user.0015_auto_20170707_1653... OK
  Applying custom_user.0016_auto_20180925_1107... OK
  Applying authentic2.0016_attribute_disabled... OK
  Applying authentic2.0017_modify_attribute_serialization... OK
  Applying authentic2.0018_auto_20170524_0842... OK
  Applying authentic2.0019_auto_20170309_1529... OK
  Applying authentic2.0020_delete_federatedid... OK
  Applying authentic2.0021_attribute_order... OK
  Applying authentic2.0022_attribute_scopes... OK
  Applying authentic2.0023_auto_20181031_0900... OK
  Applying authentic2_auth_fc.0001_initial... OK
  Applying authentic2_auth_oidc.0001_initial... OK
  Applying authentic2_auth_oidc.0002_oidcprovider_token_revocation_endpoint... OK
  Applying authentic2_auth_oidc.0003_oidcprovider_show... OK
  Applying authentic2_auth_oidc.0004_auto_20171017_1522... OK
  Applying authentic2_auth_oidc.0005_oidcprovider_slug... OK
  Applying authentic2_auth_oidc.0006_oidcprovider_claims_parameter_supported... OK
  Applying authentic2_idp_cas.0002_auto_20150410_1438... OK
  Applying authentic2_idp_cas.0003_auto_20150415_2223... OK
  Applying authentic2_idp_cas.0004_create_services... OK
  Applying authentic2_idp_cas.0005_alter_field_service_ptr... OK
  Applying authentic2_idp_cas.0006_copy_proxy_m2m... OK
  Applying authentic2_idp_cas.0007_alter_service... OK
  Applying authentic2_idp_cas.0008_alter_foreign_keys... OK
  Applying authentic2_idp_cas.0009_alter_related_models... OK
  Applying authentic2_idp_cas.0010_copy_service_ptr_id_to_old_id... OK
  Applying authentic2_idp_cas.0011_remove_old_id_restore_proxy... OK
  Applying authentic2_idp_cas.0012_copy_service_proxy_to_m2m... OK
  Applying authentic2_idp_cas.0013_delete_model_service_proxy2... OK
  Applying authentic2_idp_cas.0014_auto_20151204_1606... OK
  Applying authentic2_idp_cas.0015_auto_20170406_1825... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying authentic2_idp_oidc.0001_initial... OK
  Applying authentic2_idp_oidc.0002_auto_20170121_2346... OK
  Applying authentic2_idp_oidc.0003_auto_20170329_1259... OK
  Applying authentic2_idp_oidc.0004_auto_20170324_1426... OK
  Applying authentic2_idp_oidc.0005_authorization_mode... OK
  Applying authentic2_idp_oidc.0006_auto_20170720_1054... OK
  Applying authentic2_idp_oidc.0007_oidcclient_has_api_access... OK
  Applying authentic2_idp_oidc.0008_oidcclient_idtoken_duration... OK
  Applying authentic2_idp_oidc.0009_auto_20180313_1156... OK
  Applying authentic2_idp_oidc.0010_oidcclaim... OK
  Applying authentic2_idp_oidc.0011_auto_20180808_1546... OK
  Applying common.0001_initial... OK
  Applying common.0002_auto_20160105_1702... OK
  Applying django_rbac.0002_organizationalunit_permission_role_roleparenting... OK
  Applying django_rbac.0003_add_max_aggregate_for_postgres... OK
  Applying django_rbac.0004_auto_20150708_1337... OK
  Applying django_rbac.0005_auto_20171209_1106... OK
  Applying mellon.0001_initial... OK
  Applying nonce.0001_initial... OK
  Applying saml.0017_auto_20170710_1738... OK
  Applying sessions.0001_initial... OK
Real run
1 roles created
0 roles updated
0 ous created
1 ous updated
0 parentings created
0 parentings deleted
0 permissions created
0 permissions deleted
3 attributes created
0 attributes deleted
Success
FAILEDDestroying test database for alias 'default' ('test_authentic2_multitenant')...

====================================================================================================== FAILURES ======================================================================================================
________________________________________________________________________________________________ test_import_template ________________________________________________________________________________________________

db = None, tenant_base = '/tmp/tmpJYztgiauthentic-tenant-base'

    def test_import_template(db, tenant_base):
        call_command('create_tenant', 'authentic.example.net')
        tenant = TenantMiddleware.get_tenant_by_hostname('authentic.example.net')
        connection.set_tenant(tenant)
        call_command('import_template', '--basepath=%s' % os.path.dirname(__file__), 'data_authentic_export_site')
        content = open('%s/data_authentic_export_site.json' % os.path.dirname(__file__)).read()
>       assert byteify(export_site()) == byteify(json.loads(content))
E       assert {'ous': [{'de...e, ...}, ...]} == {'ous': [{'def...}, ...}, ...]}
E         Differing items:
E         {'ous': [{'default': True, 'description': '', 'email_is_unique': False, 'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut', ...}]} != {'ous': [{'default': True, 'description': '', 'email_is_unique': False, 'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut', ...}]}
E         {'roles': [{'description': '', 'external_id': '', 'name': 'Administrateur du r\xc3\xb4le \xc2\xab\xc2\xa0Debug eo\xc2\...: None, ...}, {'description': '', 'external_id': '', 'name': 'Administrateur des r\xc3\xb4les', 'ou': None, ...}, ...]} != {'roles': [{'attributes': [{'kind': 'string', 'name': 'is_superuser', 'value': 'true'}], 'description': '', 'external_...': 'Collectivit\xc3\xa9 par d\xc3\xa9faut', 'slug': 'default', 'uuid': '69b0a02cf58a4c71b1ae548f1375baff'}, ...}, ...]}
E         Full diff:
E         {'ous': [{'default': True,
E         'description': '',
E         'email_is_unique': False,
E         'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         'slug': 'default',
E         'username_is_unique': False,
E         -           'uuid': '569e8bd978a34581b75b19320cec7813',
E         +           'uuid': '69b0a02cf58a4c71b1ae548f1375baff',
E         'validate_emails': False}],
E         -  'roles': [{'description': '',
E         +  'roles': [{'attributes': [{'kind': 'string',
E         +                             'name': 'is_superuser',
E         +                             'value': 'true'}],
E         +             'description': '',
E         +             'external_id': '',
E         +             'name': 'Administrateur de Hobo',
E         +             'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                    'slug': 'default',
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +             'service': {'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                                'slug': 'default',
E         +                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +                         'slug': 'hobo'},
E         +             'slug': '_a2-hobo-superuser',
E         +             'uuid': '25f33158b7e2449b9a5b00dbc57bf416'},
E         +            {'attributes': [{'kind': 'string',
E         +                             'name': 'is_superuser',
E         +                             'value': 'true'}],
E         +             'description': '',
E         +             'external_id': '',
E         +             'name': 'Administrateur de Compte citoyen',
E         +             'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                    'slug': 'default',
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +             'service': {'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                                'slug': 'default',
E         +                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +                         'slug': 'portal'},
E         +             'slug': '_a2-hobo-superuser',
E         +             'uuid': '84b3b1ba76e44bcdb4fd4437c448a981'},
E         +            {'attributes': [{'kind': 'string',
E         +                             'name': 'is_superuser',
E         +                             'value': 'true'}],
E         +             'description': '',
E         +             'external_id': '',
E         +             'name': 'Administrateur de D\xc3\xa9marches',
E         +             'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                    'slug': 'default',
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +             'service': {'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                                'slug': 'default',
E         +                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +                         'slug': 'eservices'},
E         +             'slug': '_a2-hobo-superuser',
E         +             'uuid': '9054a61ccf684396b38189f1ca1ec087'},
E         +            {'attributes': [{'kind': 'string',
E         +                             'name': 'is_superuser',
E         +                             'value': 'true'}],
E         +             'description': '',
E         +             'external_id': '',
E         +             'name': 'Administrateur de Portail agent',
E         +             'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                    'slug': 'default',
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +             'service': {'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                                'slug': 'default',
E         +                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +                         'slug': 'portal-agent'},
E         +             'slug': '_a2-hobo-superuser',
E         +             'uuid': 'e6e22e5c0ca04ac0bf3b50d88eafe6d5'},
E         +            {'attributes': [{'kind': 'string',
E         +                             'name': 'is_superuser',
E         +                             'value': 'true'}],
E         +             'description': '',
E         +             'external_id': '',
E         +             'name': 'Administrateur de Passerelle',
E         +             'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                    'slug': 'default',
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +             'service': {'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         +                                'slug': 'default',
E         +                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         +                         'slug': 'passerelle'},
E         +             'slug': '_a2-hobo-superuser',
E         +             'uuid': '243f58712aa248e9b27aae669341c156'},
E         +            {'description': '',
E         'external_id': '',
E         'name': 'Administrateur du r\xc3\xb4le \xc2\xab\xc2\xa0Debug eo\xc2\xa0\xc2\xbb',
E         'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         'slug': 'default',
E         -                    'uuid': '569e8bd978a34581b75b19320cec7813'},
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         'permissions': [{'operation': {'slug': 'view'},
E         'ou': None,
E         'target': {'app_label': 'custom_user',
E         'model': 'user'},
E         'target_ct': {'app_label': 'contenttypes',
E         'model': 'contenttype'}},
E         {'operation': {'slug': 'admin'},
E         'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         'slug': 'default',
E         -                                     'uuid': '569e8bd978a34581b75b19320cec7813'},
E         +                                     'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         'target': {'name': 'Debug eo',
E         'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         'slug': 'default',
E         -                                                'uuid': '569e8bd978a34581b75b19320cec7813'},
E         ?                                                         -  -- ^^^  - ^^     ^^^^^^^^^^^^
E         +                                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         ?                                                            ^^^^^^^   ^^  +++++++++   ^^^
E         'service': None,
E         'slug': 'debug-eo',
E         'uuid': '18e7bf78dc9a432396a99f32060052ec'},
E         'target_ct': {'app_label': 'a2_rbac',
E         'model': 'role'}},
E         {'operation': {'slug': 'change'},
E         'ou': None,
E         'target': {'name': 'Administrateur du r\xc3\xb4le \xc2\xab\xc2\xa0Debug eo\xc2\xa0\xc2\xbb',
E         'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         'slug': 'default',
E         -                                                'uuid': '569e8bd978a34581b75b19320cec7813'},
E         ?                                                         -  -- ^^^  - ^^     ^^^^^^^^^^^^
E         +                                                'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         ?                                                            ^^^^^^^   ^^  +++++++++   ^^^
E         'service': None,
E         'slug': '_a2-managers-of-role-debug-eo',
E         -                                         'uuid': '89217f0a0dd74fdb996d3fd6fbc144d1'},
E         +                                         'uuid': '3049444b35874b3b9a8377ad2f10b8b6'},
E         'target_ct': {'app_label': 'a2_rbac',
E         'model': 'role'}}],
E         'service': None,
E         'slug': '_a2-managers-of-role-debug-eo',
E         -             'uuid': '89217f0a0dd74fdb996d3fd6fbc144d1'},
E         +             'uuid': '3049444b35874b3b9a8377ad2f10b8b6'},
E         {'attributes': [{'kind': 'json',
E         -                             'name': 'emails',
E         +                             'name': 'emails_to_members',
E         ?                                            +++++++++++
E         -                             'value': '[]'},
E         ?                                       ^^
E         +                             'value': 'false'},
E         ?                                       ^^^^^
E         {'kind': 'json',
E         'name': 'details',
E         'value': '""'},
E         {'kind': 'json',
E         -                             'name': 'emails_to_members',
E         ?                                            -----------
E         +                             'name': 'emails',
E         -                             'value': 'false'}],
E         ?                                       ^^^^^
E         +                             'value': '[]'}],
E         ?                                       ^^
E         'description': '',
E         'external_id': '',
E         'name': 'Debug eo',
E         'ou': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         'slug': 'default',
E         -                    'uuid': '569e8bd978a34581b75b19320cec7813'},
E         +                    'uuid': '69b0a02cf58a4c71b1ae548f1375baff'},
E         'service': None,
E         'slug': 'debug-eo',
E         'uuid': '18e7bf78dc9a432396a99f32060052ec'},
E         {'description': '',
E         'external_id': '',
E         'name': 'Administrateur',
E         'ou': None,
E         -             'parents': [{'name': 'Administrateur des utilisateurs',
E         ?                                                      ^  ^^^ ^^^^
E         +             'parents': [{'name': 'Administrateur des entit\xc3\xa9s',
E         ?                                                      ^^  ^^^^^^^ ^
E         'ou': None,
E         'service': None,
E         -                          'slug': '_a2-administrateur-des-utilisateurs',
E         ?                                                          ^  ----  --
E         +                          'slug': '_a2-administrateur-des-entites',
E         ?                                                          ^^
E         -                          'uuid': '0193477e3f2f413c9feca51f0f2b8770'},
E         +                          'uuid': 'a1ff1b3da88f47cea91e344998dfdfbf'},
E         {'name': 'Administrateur des r\xc3\xb4les',
E         'ou': None,
E         'service': None,
E         'slug': '_a2-administrateur-des-roles',
E         -                          'uuid': '7a1c92e213f2480ba0e907635b3de637'},
E         +                          'uuid': '8dd625b74cff40aa8531d7d72616550e'},
E         -                         {'name': 'Administrateur des entit\xc3\xa9s',
E         ?                                                      ^^   ^^^^^^^^
E         +                         {'name': 'Administrateur des utilisateurs',
E         ?                                                      ^  ++++ ^^^
E         'ou': None,
E         'service': None,
E         -                          'slug': '_a2-administrateur-des-entites',
E         ?                                                          ^^
E         +                          'slug': '_a2-administrateur-des-utilisateurs',
E         ?                                                          ^  ++++  ++
E         -                          'uuid': '26737d042eee400ea90c98ae75210fb9'}],
E         +                          'uuid': '4ab5effedc404fb1bcba4d21ee89b719'}],
E         'permissions': [{'operation': {'slug': 'change'},
E         'ou': None,
E         'target': {'name': 'Administrateur',
E         'ou': None,
E         'service': None,
E         'slug': '_a2-manager',
E         -                                         'uuid': 'a58074365a2740d0b34e541d90900dca'},
E         +                                         'uuid': '81a8708382bb4e8ea12ed0e172aa48b9'},
E         'target_ct': {'app_label': 'a2_rbac',
E         'model': 'role'}}],
E         'service': None,
E         'slug': '_a2-manager',
E         +             'uuid': '81a8708382bb4e8ea12ed0e172aa48b9'},
E         -             'uuid': 'a58074365a2740d0b34e541d90900dca'},
E         -            {'description': '',
E         -             'external_id': '',
E         -             'name': 'Administrateur de la collectivit\xc3\xa9 \xc2\xab\xc2\xa0Collectivit\xc3\xa9 par d\xc3\xa9faut\xc2\xa0\xc2\xbb',
E         -             'ou': None,
E         -             'permissions': [{'operation': {'slug': 'view'},
E         -                              'ou': None,
E         -                              'target': {'name': 'Collectivit\xc3\xa9 par d\xc3\xa9faut',
E         -                                         'slug': 'default',
E         -                                         'uuid': '569e8bd978a34581b75b19320cec7813'},
E         -                              'target_ct': {'app_label': 'a2_rbac',
E         -                                            'model': 'organizationalunit'}}],
E         -             'service': None,
E         -             'slug': '_a2-managers-of-default',
E         -             'uuid': 'bc3b4b21abb04a01af5ef75fcca4bcc6'},
E         {'description': '',
E         'external_id': '',
E         'name': 'Administrateur des entit\xc3\xa9s',
E         'ou': None,
E         -             'permissions': [{'operation': {'slug': 'search'},
E         ?                                                     ^ ^^^^
E         +             'permissions': [{'operation': {'slug': 'view'},
E         ?                                                     ^^ ^
E         'ou': None,
E         -                              'target': {'app_label': 'a2_rbac',
E         -                                         'model': 'organizationalunit'},
E         -                              'target_ct': {'app_label': 'contenttypes',
E         ?                                     ---                   -----------
E         +                              'target': {'app_label': 'a2_rbac',
E         ?                                                       ++++++
E         -                                            'model': 'contenttype'}},
E         ? ---                                                  ^   -------  -
E         +                                         'model': 'organizationalunit'},
E         ?                                                   ^^^^^^^^^^  +++++
E         +                              'target_ct': {'app_label': 'contenttypes',
E         +                                            'model': 'contenttype'}},
E         +                             {'operation': {'slug': 'admin'},
E         +                              'ou': None,
E         +                              'target': {'app_label': 'a2_rbac',
E         +                                         'model': 'organizationalunit'},
E         +                              'target_ct': {'app_label': 'contenttypes',
E         +                                            'model': 'contenttype'}},
E         -                             {'operation': {'slug': 'admin'},
E         ?                                                      ^^^^
E         +                             {'operation': {'slug': 'search'},
E         ?                                                     ++ ^^^
E         'ou': None,
E         'target': {'app_label': 'a2_rbac',
E         'model': 'organizationalunit'},
E         'target_ct': {'app_label': 'contenttypes',
E         'model': 'contenttype'}}],
E         'service': None,
E         'slug': '_a2-administrateur-des-entites',
E         -             'uuid': '26737d042eee400ea90c98ae75210fb9'},
E         +             'uuid': 'a1ff1b3da88f47cea91e344998dfdfbf'},
E         {'description': '',
E         'external_id': '',
E         'name': 'Administrateur des r\xc3\xb4les',
E         'ou': None,
E         'permissions': [{'operation': {'slug': 'view'},
E         'ou': None,
E         'target': {'app_label': 'custom_user',
E         'model': 'user'},
E         'target_ct': {'app_label': 'contenttypes',
E         'model': 'contenttype'}},
E         +                             {'operation': {'slug': 'view'},
E         +                              'ou': None,
E         +                              'target': {'app_label': 'a2_rbac',
E         +                                         'model': 'organizationalunit'},
E         +                              'target_ct': {'app_label': 'contenttypes',
E         +                                            'model': 'contenttype'}},
E         +                             {'operation': {'slug': 'admin'},
E         +                              'ou': None,
E         +                              'target': {'app_label': 'a2_rbac',
E         +                                         'model': 'role'},
E         +                              'target_ct': {'app_label': 'contenttypes',
E         +                                            'model': 'contenttype'}},
E         {'operation': {'slug': 'search'},
E         'ou': None,
E         'target': {'app_label': 'a2_rbac',
E         'model': 'organizationalunit'},
E         'target_ct': {'app_label': 'contenttypes',
E         -                                            'model': 'contenttype'}},
E         -                             {'operation': {'slug': 'admin'},
E         -                              'ou': None,
E         -                              'target': {'app_label': 'a2_rbac',
E         -                                         'model': 'role'},
E         -                              'target_ct': {'app_label': 'contenttypes',
E         'model': 'contenttype'}}],
E         'service': None,
E         'slug': '_a2-administrateur-des-roles',
E         -             'uuid': '7a1c92e213f2480ba0e907635b3de637'},
E         +             'uuid': '8dd625b74cff40aa8531d7d72616550e'},
E         {'description': '',
E         'external_id': '',
E         'name': 'Administrateur des utilisateurs',
E         'ou': None,
E         -             'permissions': [{'operation': {'slug': 'search'},
E         ?                                                     ^ ^^^^
E         +             'permissions': [{'operation': {'slug': 'view'},
E         ?                                                     ^^ ^
E         'ou': None,
E         'target': {'app_label': 'a2_rbac',
E         'model': 'organizationalunit'},
E         'target_ct': {'app_label': 'contenttypes',
E         'model': 'contenttype'}},
E         {'operation': {'slug': 'admin'},
E         'ou': None,
E         'target': {'app_label': 'custom_user',
E         'model': 'user'},
E         'target_ct': {'app_label': 'contenttypes',
E         +                                            'model': 'contenttype'}},
E         +                             {'operation': {'slug': 'search'},
E         +                              'ou': None,
E         +                              'target': {'app_label': 'a2_rbac',
E         +                                         'model': 'organizationalunit'},
E         +                              'target_ct': {'app_label': 'contenttypes',
E         'model': 'contenttype'}}],
E         'service': None,
E         'slug': '_a2-administrateur-des-utilisateurs',
E         -             'uuid': '0193477e3f2f413c9feca51f0f2b8770'}]}
E         +             'uuid': '4ab5effedc404fb1bcba4d21ee89b719'}]}

tests_authentic/test_hobo_deploy.py:454: AssertionError

#26 Updated by Benjamin Dauvergne 4 months ago

Christophe Siraut a écrit :

J'ai simplifié le patch et les tests.

Les tests coté passerelle passent (avec un json minimaliste); ceux de authentic échouent (avec un json complet issu d'un export). Pour commencer le dictionnaire renvoyé par authentic2.data_transfer.export_site contient des entrées unicode tandis que celles de passerelle sont toutes en bytes. J'ai utilisé byteify() pour contourner le problème coté a2 mais je ne me rend pas compte s'il y a lieu de corriger quelque chose en amont.

Ensuite voici les différences observées entre authentic2.data_transfer.export_site() et les données présentes dans le template json. Il y a, outre le problème d'encodage, des uuid qui diffèrent, mais je ne me rend pas compte si le json que je lui donne à manger est représentatif de ce qu'on veut faire (ou si les rôles de bases devraient être exclus du usecase)

Pour moi il y a juste le problème de l'uuid il ne peut pas y avoir de problème d'encodage, coté a2 on exporte soit de l'unicode soit des str ascii qui vont matcher automatiquement l'unicode, une possibilité ce serait de virer les uuid avant comparaison des deux cotés.

#27 Updated by Benjamin Dauvergne 4 months ago

Benjamin Dauvergne a écrit :

Pour moi il y a juste le problème de l'uuid il ne peut pas y avoir de problème d'encodage, coté a2 on exporte soit de l'unicode soit des str ascii qui vont matcher automatiquement l'unicode, une possibilité ce serait de virer les uuid avant comparaison des deux cotés.

Il y a l'ordre de certains listes qui semble différer aussi (liste des permissions), on peut vraisemblablement normaliser coté a2 (en triant).

#28 Updated by Christophe Siraut 4 months ago

le résultat de la comparaison en supprimant les clés uuid et en triant les listes : https://pad.libre-entreprise.org/p/import-template

#29 Updated by Benjamin Dauvergne 4 months ago

Christophe Siraut a écrit :

le résultat de la comparaison en supprimant les clés uuid et en triant les listes : https://pad.libre-entreprise.org/p/import-template

Encore d'autres liste qui ne sont pas triés dans le même ordre, parents, attributes, ça risque de devenir un peu chiant :/

#30 Updated by Christophe Siraut 4 months ago

  • Status changed from En cours to Solution proposée

tests et build réussi.

#31 Updated by Christophe Siraut 3 months ago

Rebasé sur le master du jour.

#32 Updated by Benjamin Dauvergne 3 months ago

  • Status changed from Solution proposée to Solution validée

Ack.

#33 Updated by Christophe Siraut 3 months ago

  • Status changed from Solution validée to Résolu (à déployer)
commit 03a2aa8ed49774adee37ba78d950ed5fd6944a5e (HEAD -> master, origin/wip/14630-Avoir-un-import-template-generique, origin/master, origin/HEAD, wip/14630-Avoir-un-import-template-generique)
Author: Christophe Siraut <csiraut@entrouvert.com>
Date:   Thu Jan 10 21:34:51 2019 +0100

    agent: add generic import_template management command (#14630)

#34 Updated by Frédéric Péters 3 months ago

  • Status changed from Résolu (à déployer) to Solution déployée

#35 Updated by Nicolas Roche 2 months ago

  • Related to Bug #32469: mauvais chemin par défaut lors de l'import des templates added

Also available in: Atom PDF