Development #5244
upgrade for Django 1.7
0%
Description
We need to start targeting Django 1.6 without losing compatibility with Django 1.5
Fichiers
Révisions associées
[django-1.6] adapt to API change on EmailValidator
keep local implementation backward compatible with Django < 1.6
refs #5244
[django-1.6] middleware: do not store set() object in sessions only lists
refs #5244
[django-1.6] use ATOMIC_REQUESTS setting instead of TransactionMiddleware
Also remove all use of commit_on_success decorator which is useless
since all requests are atomic.
refs #5244
[django-1.6] fix import path of FieldDoesNotExist exception
refs #5244
[django-1.6] adapt to API change on EmailValidator
keep local implementation backward compatible with Django < 1.6
refs #5244
[django-1.6] middleware: do not store set() object in sessions only lists
refs #5244
[django-1.6] use ATOMIC_REQUESTS setting instead of TransactionMiddleware
Also remove all use of commit_on_success decorator which is useless
since all requests are atomic.
refs #5244
[django-1.6] authentication backends import path must match the canonical module.__class__
refs #5244
[django-1.6] LDAPUser application cannot be deduced without a Meta.app_label
refs #5244
[django-1.6] add default value to all BooleanField missing it
refs #5244
Added tox as a test-runner.
It will build different virtualenvs for each target.
getlasso.sh is somewhat hacky, but until lasso is installable through pip this is the workaround.
License: MIT
refs #5244
Create a base_no_sekizai.html base template for 404 and 500 templates as they are used by Django tests which do not install django-sekizai
refs #5244
Add setting A2_VALIDATE_EMAIL_DOMAIN to completely disable email domain checking
refs #5244
Add default value to test_setting to accomodate needs of Django tests
refs #5244
Run tox as part of the continuous integration script, stop the script on any error
refs #5244
Add Django 1.7 environment to tox configuration
fixes #5244
[django-1.7] Declare authentic2 compatible with django 1.7
refs #5244
[django-1.7] Rename all migrations directories to south_migrations and depends on south >= 1.0.0
refs #5244
[django-1.7] User profiles were deprecated in django 1.5, partially remove the functionnality from our copy of AbstractUser
refs #5244
[django-1.7] Use new application config ready() method to fix user models
refs #5244
[django-1.7] Use application configuration to rename the SAML 2.0 idp application and prevent name collision
refs #5244
[django-1.7] Use allow lazy to apply string tranformation to translatable string in models definitions
refs #5244
[django-1.7] Replace django-registration by django-registration-redux to get Django 1.7 compatibility
refs #5244
[django-1.6] Add missing default value to AttributePolicy.enabled field
refs #5244
[django-1.6] Not settings Meta.fields or Meta.exclude has been deprecated
refs #5244
[django-1.7] Natural primary key support have been added to Django 1.7, we only need natural generic foreign key support now
refs #5244
[django-1.7] Prevent Django 1.7 showing a warning about test suites initialized before Django 1.6
refs #5244
[django-1.7] Add migrations for Django 1.7
refs #5244
[django-1.7] Declare authentic2 compatible with django 1.7
refs #5244
[django-1.7] User profiles were deprecated in django 1.5, partially remove the functionnality from our copy of AbstractUser
refs #5244
[django-1.7] Use new application config ready() method to fix user models
refs #5244
[django-1.7] Use application configuration to rename the SAML 2.0 idp application and prevent name collision
refs #5244
[django-1.7] Use allow lazy to apply string tranformation to translatable string in models definitions
refs #5244
[django-1.6] Add missing default value to AttributePolicy.enabled field
refs #5244
[django-1.6] Not settings Meta.fields or Meta.exclude has been deprecated
refs #5244
[django-1.7] Natural primary key support have been added to Django 1.7, we only need natural generic foreign key support now
refs #5244
[django-1.7] Prevent Django 1.7 showing a warning about test suites initialized before Django 1.6
refs #5244
Historique
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- Fichier 0001-django-1.6-fix-import-path-of-FieldDoesNotExist-exce.patch 0001-django-1.6-fix-import-path-of-FieldDoesNotExist-exce.patch ajouté
- Fichier 0002-django-1.6-adapt-to-API-change-on-EmailValidator.patch 0002-django-1.6-adapt-to-API-change-on-EmailValidator.patch ajouté
- Patch proposed changé de Non à Oui
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- Fichier 0003-django-1.6-middleware-do-not-store-set-object-in-ses.patch 0003-django-1.6-middleware-do-not-store-set-object-in-ses.patch ajouté
- Fichier 0004-django-1.6-use-ATOMIC_REQUESTS-setting-instead-of-Tr.patch 0004-django-1.6-use-ATOMIC_REQUESTS-setting-instead-of-Tr.patch ajouté
- Fichier 0005-django-1.6-authentication-backends-import-path-must-.patch 0005-django-1.6-authentication-backends-import-path-must-.patch ajouté
- Fichier 0006-django-1.6-LDAPUser-application-cannot-be-deduced-wi.patch 0006-django-1.6-LDAPUser-application-cannot-be-deduced-wi.patch ajouté
Autres patchs pour Django 1.6.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- Sujet changé de upgrade for Django 1.6 à upgrade for Django 1.7
Now we target Django 1.7.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- https://django.readthedocs.org/en/latest/releases/1.6.html
- https://django.readthedocs.org/en/latest/releases/1.7.html
Maybe faster to review the deprecation timeline: https://django.readthedocs.org/en/latest/internals/deprecation.html#deprecation-removed-in-1-6
After a fast reading of the the deprecation timeline I see these important points to review:- BooleanField does not have default=False by default https://django.readthedocs.org/en/latest/releases/1.6.html#booleanfield-no-longer-defaults-to-false
- URL quoting in reverse() https://django.readthedocs.org/en/latest/releases/1.6.html#quoting-in-reverse
- execute_manager() has been removed:
tests/integration/saml2/idp_manage.py: execute_manager(settings) tests/integration/saml2/sp_manage.py: execute_manager(settings)
HttpRequest.raw_post_data
was renamed.body
authentic2/saml/common.py:61: return request.raw_post_data
- HttReponse.__init__ argument mimetype was renamed content_type
authentic2/idp/idp_openid/views.py:216: }, context_instance=RequestContext(request), mimetype=YADIS_CONTENT_TYPE) authentic2/idp/saml/saml2_endpoints.py:105: mimetype='text/xml') authentic2/saml/common.py:177: return HttpResponse(profile.msgBody, mimetype = 'text/xml') authentic2/saml/common.py:216: return HttpResponse(profile.msgBody, mimetype = 'text/xml') authentic2/saml/common.py:636: return HttpResponse(content, mimetype = "text/xml")
Transaction management should be reviewed too.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
I wanted to evaluate this, so I created a branch of this on Github:
https://github.com/grigi/authentic2
Fisrtly I wanted to get a full test suite running for both Django 1.5 and 1.6, to ensure that these patches don't obviously break something.
Due to the invasive nature of authentic2, I ensured that all tests for Django, 3'rd party apps and authentic2 ran.
Commit: Added tox to run tests for both Djang1.5 and Django1.6
1.5: 3 failures, 12 errors
1.6: 103 errors
0001 & 0002 failed to apply.
Commit: applied 0003-django-1.6-middleware-do-not-store-set-object-in-ses.patch
1.5: 3 failures
1.6: 21 failures, 6 errors
0004 & 0005 failed to apply
Commit: applied 0006-django-1.6-LDAPUser-application-cannot-be-deduced-wi.patch
No change in test results
Experiment
Of the 3 failures in 1.5, 2 are to do with authentic2.backends.ModelBackend, 1 with django-registration
So I Temporarily Changed AUTHENTICATION_BACKENDS to use django.contrib.auth.backends.ModelBackend instead
1.5: 1 failure
1.6: 1 failure, 2 errors
This means there is something not right in authentic2.backends.ModelBackend, is this needed?
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
You should maybe consider getting rid of django-registration as a dependency, as it doesn't work with Django>1.5.
There is a ticket on the bitbucket page to add support for 1.6, but the maintainer essentially quit maintaining it, and nobody else took ownership.
I made a mistake with my test runner for Django>=1.6 where django-registration test filenames don't conform to the new pattern match, so you have to do a -p init.py for registration ONLY.
There is a package called django-registration2, but it seems to have been forked from django-registration very long ago, and is missing much of the new features :-(
The general consensus on the net is to migrate over to django-allauth, but I haven't evaluated if it uses all the features you use.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
Commit: applied 0001-django-1.6-fix-import-path-of-FieldDoesNotExist-exce.patch
1.5: 2 failures
1.6: 21 failures, 6 errors
I cannot apply 0002 manually. It looks like the code got overhauled since.
I cannot apply 0004 cleanly, and the existing is still valid through to Django 1.7. Whereas it will break 1.5 compatibility. So I'm proposing skipping it.
Commit: manually applied 0005-django-1.6-authentication-backends-import-path-must-.patch
1.5: 2 failures
1.6: 4 failures, 2 errors
I applied all the patches here that I could, and it definitely helped 1.6 compatibility and actually helped 1.5 as well.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
- Fichier 0007-added-default-False-to-all-BooleanFields-that-didn-t.patch 0007-added-default-False-to-all-BooleanFields-that-didn-t.patch ajouté
- Fichier 0008-changed-.raw_post_data-to-.body-as-per-django1.4-dep.patch 0008-changed-.raw_post_data-to-.body-as-per-django1.4-dep.patch ajouté
- Fichier 0009-changed-mimetype-to-content_type-as-per-django1.5-de.patch 0009-changed-mimetype-to-content_type-as-per-django1.5-de.patch ajouté
Commit: added default=False to all BooleanFields that didn't have an explicit default
1.5: 2 failures
1.6: 4 failures, 1 error
Commit: changed .raw_post_data to .body as per django1.4 deprecation rules.
1.5: 2 failures
1.6: 4 failures, 1 error
Commit: changed mimetype to content_type as per django1.5 deprecation rules.
1.5: 2 failures
1.6: 4 failures, 1 error
The tests under tests/integration/saml2/ isn't usable as they are now, and should probably be rewritten to use a LiveServerTestCase and test-specific fixtures.
I also found no more things to do based on the deprecation info from Django.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
- Fichier 0001-applied-0003-django-1.6-middleware-do-not-store-set-.patch 0001-applied-0003-django-1.6-middleware-do-not-store-set-.patch ajouté
- Fichier 0002-applied-0006-django-1.6-LDAPUser-application-cannot-.patch 0002-applied-0006-django-1.6-LDAPUser-application-cannot-.patch ajouté
- Fichier 0003-applied-0001-django-1.6-fix-import-path-of-FieldDoes.patch 0003-applied-0001-django-1.6-fix-import-path-of-FieldDoes.patch ajouté
- Fichier 0004-manually-applied-0005-django-1.6-authentication-back.patch 0004-manually-applied-0005-django-1.6-authentication-back.patch ajouté
- Fichier 0005-added-default-False-to-all-BooleanFields-that-didn-t.patch 0005-added-default-False-to-all-BooleanFields-that-didn-t.patch ajouté
- Fichier 0006-changed-.raw_post_data-to-.body-as-per-django1.4-dep.patch 0006-changed-.raw_post_data-to-.body-as-per-django1.4-dep.patch ajouté
- Fichier 0007-changed-mimetype-to-content_type-as-per-django1.5-de.patch 0007-changed-mimetype-to-content_type-as-per-django1.5-de.patch ajouté
I evaluated all the failing tests, this is my findings:
Django 1.5:
2 failiures relates to 'authentic2.backends.models_backend.ModelBackend'
Django 1.6:
3 failiures relates to 'authentic2.backends.models_backend.ModelBackend'
1 failiure is because you are overriding LOGIN_REDIRECT_URL, and the test case test the default, hence an expected failure in this case
1 error is to do with django.contrib.messages and I can't see why it is failing. None of the authentic2 code is causing it to fail.
The tests failing due to 'authentic2.backends.models_backend.ModelBackend' actually seems correct behaviour since you wrapped User in a ProxyUser class. I'm not sure why that handler exists, as it doesn't seem to do much above djangos modelBackend, and changing it doesn't cause any regressions that I can notice when using the system.
I think using this methodology of running 3rd party tests has exhausted updating the code to port to django 1.6.
Using it as it is now, it seems to work pretty well, except for django-registration (which I don't need in my installation).
Could you please evaluate the changes I did to fix Django 1.6 and apply as much of them as you are comfortable to avoid the issue of the patches getting even more out of date?
I re-generated 7 commits as patch files, that should apply cleanly to the current version, and doesn't break 1.5 compat, but essentially fixes 1.6 except for removing registration, which I think is outside the scope of this.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
- Fichier 0001-Added-tox-as-a-test-runner-it-will-build-different-v.patch 0001-Added-tox-as-a-test-runner-it-will-build-different-v.patch ajouté
For informational purposes only: Attached is a patch for getting tox up and running for this.
tox is nice for when you need to run tests for multiple different installations, and in this case django1.6 and django1.7.
Note: getlasso.sh is very hacky and should probably be changed.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
Could you add a "License: MIT" line to your patchs ? We only accept contributions with a BSD-like license (I should add the policy to the README file).
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
Just to be clear the License line must be added to the end of your commit message (last line of it_.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
- Fichier 0001-Added-tox-as-a-test-runner.patch 0001-Added-tox-as-a-test-runner.patch ajouté
- Fichier 0001-applied-0003-django-1.6-middleware-do-not-store-set-.patch 0001-applied-0003-django-1.6-middleware-do-not-store-set-.patch ajouté
- Fichier 0002-applied-0006-django-1.6-LDAPUser-application-cannot-.patch 0002-applied-0006-django-1.6-LDAPUser-application-cannot-.patch ajouté
- Fichier 0003-applied-0001-django-1.6-fix-import-path-of-FieldDoes.patch 0003-applied-0001-django-1.6-fix-import-path-of-FieldDoes.patch ajouté
- Fichier 0004-manually-applied-0005-django-1.6-authentication-back.patch 0004-manually-applied-0005-django-1.6-authentication-back.patch ajouté
- Fichier 0005-added-default-False-to-all-BooleanFields-that-didn-t.patch 0005-added-default-False-to-all-BooleanFields-that-didn-t.patch ajouté
- Fichier 0006-changed-.raw_post_data-to-.body-as-per-django1.4-dep.patch 0006-changed-.raw_post_data-to-.body-as-per-django1.4-dep.patch ajouté
- Fichier 0007-changed-mimetype-to-content_type-as-per-django1.5-de.patch 0007-changed-mimetype-to-content_type-as-per-django1.5-de.patch ajouté
Ok, I added License: MIT to each commit.
I also supplied a slightly less flaky getlasso.sh to the 0001-Added-tox-as-a-test-runner.patch
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
Hi
Is there any particular reason that this issue is being held up?
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
Sorry I lack time to review it currently, I'll try to do it soon.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
I applied your patches and mine on this branch : http://repos.entrouvert.org/authentic.git/log/?h=wip/5244-django-17-compatibility could you verify it works on your side and the I will merge it. The remaining failure I see with tests from django.contrib.* applications are all related to to assumptions from Django which are wrong, for example that only if CUSTOM_USER_MODEL is set, authentication backend returns non-User models. Authentic2 LDAP backend does that, it returns a proxy-model to the classic User model, which is not an error.
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
- Fichier 0001-Atomic-settings.patch 0001-Atomic-settings.patch ajouté
I tested it on our implementation using both Django 1.5 and 1.6, and other than a minor settings change needed (attached patch) everything works nicely :-)
As we don't depend on registration, I wasn't affected by registration not supporting Django>1.5
The messages tests "test_middleware_disabled" are failing where sekizai complains about middleware not correctly configured, which is to spec of sekizai, so I see no problem.
Thanks, looks good :-)
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- Statut changé de Nouveau à Résolu (à déployer)
- % réalisé changé de 0 à 100
Appliqué par commit 58d827d4ec44ca370dce3c139039686ce618e921.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- Statut changé de Résolu (à déployer) à Fermé
- % réalisé changé de 100 à 0
Ok, I fixed my patch about ATOMIC_REQUESTS (I misread the documentation I think) with yours, I fixed all boolean field missing a default value, I added integration to our jenkins jobs to run tox on each commit, a django 1.7 section to tox.ini. It's pushed to master, thank your for your help on this ticket.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
- Statut changé de Fermé à En cours
I re-open this ticket as there are still some incompatibilities for running with django 1.7, like django 1.7 migrations.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
I made a new serie of patches to fully support Django 1.7, would you test them ?
http://repos.entrouvert.org/authentic.git/log/?h=wip/django-1.7-migrations
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
- Fichier 0001-Documentation-to-upgrade-to-2.2-bump-to-2.2.0dev.patch 0001-Documentation-to-upgrade-to-2.2-bump-to-2.2.0dev.patch ajouté
- Fichier 0002-Makemigrations-these-seem-valid.patch 0002-Makemigrations-these-seem-valid.patch ajouté
- Fichier 0003-DONTAPPLY-There-is-something-wrong-with-the-Attribut.patch 0003-DONTAPPLY-There-is-something-wrong-with-the-Attribut.patch ajouté
Wow, that was quick!
I tested upgrading from south to django1.7, and it seems you MUST finish applying all the migrations before upgrading to 1.7, then upgrade to 1.7 and re-run migrate.
You should probably add that to the docs.
I attached updated docs.
Then I also noticed that migrate complained I had unmigrated models, so I ran makemigrations, (patch 2)
and found something interesting (not in a good way).
There is something wrong with the Attribute.kind field enum, it always points to some proxymodel instance, hence is unstable. You have a better idea as to what REALLY needs to be done here.
Demonstration is on patch3, or re-run makemigrations, and it will keep on changing that enum :-(
I then ran tests and have some failiures:
django.db.utils.OperationalError: no such table: auth_customuser
django.db.utils.OperationalError: no such table: contenttypes_concretemodel
excluding django.contrib.auth and django.contrib.contenttypes resolves that issue, but I'm actually quite interested in the django.contrib.auth tests.
Mis à jour par Benjamin Dauvergne il y a plus de 9 ans
I will not merge the Django 1.7 migrations until we decide that Django 1.7 is mandatory, as the move for every deployment will be one way and we cannot manage two lines of development. Until this time Django 1.7 support will be experimental.
As for the customizable choices varying at each makemigrations it's a known limitations of the new migration system1; the only fix will be to move all variable choices in forms and not on models; default choices on models will be empty.
[1]: https://code.djangoproject.com/ticket/22837 and https://code.djangoproject.com/ticket/23581
Mis à jour par Nickolas Grigoriadis il y a plus de 9 ans
Ah, ok. that explains that.
Yes, the upgrade to Django 1.7 can be quite disruptive, which is why I originally only tried to get 1.6 to work.
I really only needed Django 1.6 support, because Django 1.5 is now formally not maintained any more, and won't get more security updates, but 1.6 will still for about half a year.
If you can maybe apply some of the fixes from that branch that is not in master yet? The django-registration-redux which we need for Django 1.6 would allow us to do the upgrade a little more incrementally?
e.g. Commits:
#5d006795 #39c63d79 #62c9cab8 #86bd809b #14f969e4
look safe for me to apply?
And some documentation that you may need to "pip uninstall django-registration" on upgrade?
Mis à jour par Benjamin Dauvergne il y a environ 9 ans
- Statut changé de En cours à Solution déployée
- Patch proposed changé de Oui à Non
Everything has been pushed but Django 1.7 initial migrations.
Mis à jour par Benjamin Dauvergne il y a environ 9 ans
- Statut changé de Solution déployée à Fermé
[django-1.6] fix import path of FieldDoesNotExist exception
refs #5244