Project

General

Profile

Development #69090

Requête web beaucoup trop longue sur /api/cards

Added by Pierre Ducroquet 3 months ago. Updated 3 months ago.

Status:
Fermé
Priority:
Normal
Target version:
-
Start date:
14 September 2022
Due date:
% Done:

0%

Estimated time:
Patch proposed:
Yes
Planning:
No

Description

Observé sur toulouse preprod, #69080 : la liste des 501 éléments prend 10 secondes à être générée.

https://demarches-montoulouse.cutm-publik-preprod.nfrance.com/api/cards/reservation-activite-sportive/list?full=on

Plusieurs pistes à suivre:
- réduire le volume du full=on (on passe de 10s à 700ms a priori)
- comprendre où sont passées les 10 secondes


Files


Related issues

Related to w.c.s. - Development #69108: api: performance du endpoint list, permettre d'exclure les evolutionsSolution déployée15 September 2022

Actions
Related to w.c.s. - Development #69109: api: performance du endpoint list, charger les parts de manière paresseuseSolution déployée15 September 2022

Actions
Related to w.c.s. - Development #69126: api: performance, l'appel à preprocess_struct_time dans JSONEncoder.encode est coûteuxNouveau15 September 2022

Actions

Associated revisions

Revision 0296a9e8 (diff)
Added by Benjamin Dauvergne 3 months ago

api: does not write to disk during form/card listing (#69090)

The call to increment_count() is only beneficial to the JSON file export
case.

History

#1

Updated by Benjamin Dauvergne 3 months ago

  • Description updated (diff)
#2

Updated by Benjamin Dauvergne 3 months ago

En fait on perd du temps à écrire des fichiers sur le disque dans AfterJob.increment_count()

In [87]: def f():
    ...:     items = list(carddef.data_class().select([Equal('fbod24ee8d0-8da6-4562-a8e4-f0eca4e0bd26', False)]))
    ...:     carddef.data_class().load_all_evolutions(items)
    ...:     data = JsonFileExportAfterJob(carddef).create_json_export(items, user=None, anonymise=False, digest_key='default', include_evolution=False, include_roles=False, include_files=False, include_unnamed_
    ...: fields=False)
    ...:     return len(json.dumps(data, cls=JSONEncoder))
    ...: 

In [88]: cProfile.run('f()',  sort=SortKey.CUMULATIVE)
         817242 function calls (737463 primitive calls) in 9.396 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    9.396    9.396 {built-in method builtins.exec}
        1    0.001    0.001    9.396    9.396 <string>:1(<module>)
        1    0.000    0.000    9.395    9.395 <ipython-input-87-dd7a1680e2f2>:1(f)
        1    0.006    0.006    8.448    8.448 management.py:4270(create_json_export)
      495    0.002    0.000    8.190    0.017 afterjobs.py:78(increment_count)
      495    0.023    0.000    8.188    0.017 storage.py:830(store)
      495    0.004    0.000    6.427    0.013 storage.py:80(atomic_write)
      495    0.017    0.000    6.423    0.013 storage.py:84(doit)
      495    0.002    0.000    2.785    0.006 tempfile.py:435(mkstemp)
      495    0.009    0.000    2.780    0.006 tempfile.py:378(_mkstemp_inner)
      495    2.725    0.006    2.725    0.006 {built-in method posix.open}
      495    1.456    0.003    1.456    0.003 {built-in method posix.fsync}
      495    1.093    0.002    1.093    0.002 {built-in method posix.rename}
      495    0.868    0.002    0.868    0.002 {built-in method posix.fchmod}
      495    0.854    0.002    0.854    0.002 {built-in method posix.utime}
3704/3703    0.002    0.000    0.843    0.000 sql.py:629(f)
        1    0.005    0.005    0.705    0.705 sql.py:2585(load_all_evolutions)
     3966    0.016    0.000    0.655    0.000 sql.py:105(pickle_loads)
     1986    0.008    0.000    0.653    0.000 sql.py:2568(_row2evo)
     3967    0.614    0.000    0.633    0.000 {method 'load' of '_pickle.Unpickler' objects}
      495    0.001    0.000    0.369    0.001 locket.py:122(__enter__)
      495    0.004    0.000    0.368    0.001 locket.py:105(acquire)
      495    0.002    0.000    0.362    0.001 locket.py:160(acquire)
      495    0.001    0.000    0.281    0.001 locket.py:126(__exit__)
      495    0.003    0.000    0.281    0.001 locket.py:117(release)
      495    0.004    0.000    0.276    0.001 locket.py:173(release)
      990    0.259    0.000    0.259    0.000 {built-in method fcntl.flock}
      495    0.025    0.000    0.249    0.001 formdata.py:1342(get_json_export_dict)
      991    0.227    0.000    0.234    0.000 {built-in method io.open}
      495    0.168    0.000    0.168    0.000 {method 'close' of '_io.BufferedWriter' objects}
      990    0.003    0.000    0.152    0.000 genericpath.py:16(exists)
      990    0.149    0.000    0.149    0.000 {built-in method posix.stat}
      495    0.146    0.000    0.146    0.000 {method 'close' of '_io.TextIOWrapper' objects}
      495    0.002    0.000    0.138    0.000 locket.py:44(_lock_file_blocking)
        1    0.000    0.000    0.134    0.134 sql.py:2294(select)
      496    0.000    0.000    0.134    0.000 sql.py:2227(select_iterator)
      495    0.002    0.000    0.126    0.000 locket.py:59(_unlock_file)
        1    0.000    0.000    0.108    0.108 __init__.py:183(dumps)
        1    0.001    0.001    0.108    0.108 misc.py:603(encode)
      496    0.000    0.000    0.102    0.000 sql.py:2243(retrieve)
      496    0.001    0.000    0.102    0.000 sql.py:2174(get_objects_iterator)
  78294/1    0.060    0.000    0.084    0.084 misc.py:586(preprocess_struct_time)
        1    0.000    0.000    0.084    0.084 misc.py:593(<listcomp>)
      495    0.006    0.000    0.078    0.000 formdata.py:1262(get_display_name)
      495    0.009    0.000    0.072    0.000 sql.py:2863(_row2ob)
        2    0.066    0.033    0.067    0.034 {method 'execute' of 'psycopg2.extensions.cursor' objects}
      990    0.003    0.000    0.061    0.000 formdata.py:804(get_url)
      990    0.004    0.000    0.057    0.000 carddef.py:121(get_url)
      990    0.004    0.000    0.053    0.000 publisher.py:164(get_backoffice_url)
      495    0.021    0.000    0.051    0.000 sql.py:2392(_row2obdata)
      495    0.002    0.000    0.048    0.000 functional.py:202(__wrapper__)
      495    0.002    0.000    0.046    0.000 functional.py:109(__init__)
      990    0.002    0.000    0.046    0.000 formdata.py:1332(get_json_dict)
      495    0.014    0.000    0.044    0.000 functional.py:125(__prepare_class__)
      990    0.010    0.000    0.044    0.000 parse.py:516(urljoin)
      990    0.035    0.000    0.044    0.000 formdata.py:1287(get_json_data_dict)
     2483    0.018    0.000    0.036    0.000 {method 'fetchone' of 'psycopg2.extensions.cursor' objects}
   301842    0.033    0.000    0.033    0.000 {built-in method builtins.isinstance}
      495    0.008    0.000    0.029    0.000 locket.py:67(lock_file)
    63899    0.023    0.000    0.026    0.000 {built-in method builtins.hasattr}
        1    0.000    0.000    0.023    0.023 encoder.py:182(encode)
        1    0.020    0.020    0.022    0.022 encoder.py:204(iterencode)
      495    0.003    0.000    0.022    0.000 functional.py:190(__mod__)
      495    0.003    0.000    0.022    0.000 posixpath.py:372(abspath)
      495    0.001    0.000    0.021    0.000 {built-in method builtins.next}
     1980    0.008    0.000    0.021    0.000 parse.py:369(urlparse)
      495    0.001    0.000    0.021    0.000 storage.py:826(storage_dumps)
      495    0.001    0.000    0.021    0.000 formdata.py:807(get_backoffice_url)
      495    0.018    0.000    0.020    0.000 {built-in method _pickle.dumps}
      495    0.003    0.000    0.020    0.000 tempfile.py:284(__next__)
      991    0.006    0.000    0.019    0.000 storage.py:450(get_objects_dir)
      495    0.001    0.000    0.019    0.000 functional.py:155(__text_cast)
      990    0.001    0.000    0.018    0.000 _json.py:164(typecast_json)
     5977    0.012    0.000    0.018    0.000 publisher.py:50(find_class)
      495    0.004    0.000    0.018    0.000 __init__.py:41(gettext)
      990    0.001    0.000    0.018    0.000 __init__.py:299(loads)
      495    0.005    0.000    0.017    0.000 os.py:1019(fdopen)
39176/37691    0.011    0.000    0.017    0.000 sql.py:466(str_encode)
     2477    0.011    0.000    0.016    0.000 posixpath.py:71(join)
      990    0.002    0.000    0.016    0.000 decoder.py:332(decode)
      495    0.002    0.000    0.015    0.000 storage.py:909(update_indexes)
      495    0.002    0.000    0.013    0.000 tempfile.py:287(<listcomp>)
      495    0.004    0.000    0.013    0.000 formdata.py:1226(get_last_update_time)
      495    0.013    0.000    0.013    0.000 {method 'mro' of 'type' objects}
    46109    0.012    0.000    0.012    0.000 {method 'get' of 'dict' objects}
      495    0.004    0.000    0.012    0.000 formdata.py:708(get_visible_status)
      495    0.002    0.000    0.012    0.000 __init__.py:78(gettext)
      495    0.012    0.000    0.012    0.000 {method 'flush' of '_io.BufferedWriter' objects}
     3960    0.004    0.000    0.011    0.000 random.py:344(choice)
      990    0.011    0.000    0.011    0.000 decoder.py:343(raw_decode)
      495    0.004    0.000    0.010    0.000 locket.py:76(_create_lock_file)
      495    0.004    0.000    0.010    0.000 trans_real.py:343(gettext)
     6930    0.008    0.000    0.010    0.000 parse.py:111(_coerce_args)
      495    0.006    0.000    0.009    0.000 posixpath.py:334(normpath)
      990    0.005    0.000    0.009    0.000 formdata.py:685(get_status)
      495    0.002    0.000    0.009    0.000 posixpath.py:60(isabs)
     3962    0.007    0.000    0.009    0.000 posixpath.py:41(_get_sep)
    13902    0.004    0.000    0.009    0.000 sql.py:2571(<genexpr>)
      990    0.002    0.000    0.008    0.000 parse.py:486(urlunparse)
     1980    0.004    0.000    0.008    0.000 parse.py:434(urlsplit)
    14451    0.007    0.000    0.007    0.000 {built-in method builtins.getattr}
      495    0.003    0.000    0.007    0.000 weakref.py:162(__setitem__)
     3960    0.005    0.000    0.007    0.000 random.py:238(_randbelow_with_getrandbits)
      495    0.003    0.000    0.007    0.000 _bootlocale.py:33(getpreferredencoding)
     1485    0.002    0.000    0.006    0.000 publisher.py:148(get_frontoffice_url)
     4464    0.003    0.000    0.006    0.000 monkeypatch.py:46(get_publisher)
     2611    0.006    0.000    0.006    0.000 {method 'timetuple' of 'datetime.datetime' objects}
      495    0.002    0.000    0.005    0.000 formdata.py:810(get_api_url)
      495    0.002    0.000    0.005    0.000 gettext.py:496(gettext)
      990    0.003    0.000    0.004    0.000 parse.py:497(urlunsplit)
      495    0.004    0.000    0.004    0.000 weakref.py:192(get)
      495    0.002    0.000    0.004    0.000 posixpath.py:150(dirname)
      495    0.004    0.000    0.004    0.000 {built-in method _locale.nl_langinfo}
     1485    0.002    0.000    0.004    0.000 publisher.py:1012(get_cfg)
    11385    0.003    0.000    0.003    0.000 {built-in method builtins.setattr}
      495    0.003    0.000    0.003    0.000 locket.py:131(__init__)
     6437    0.003    0.000    0.003    0.000 {method 'startswith' of 'str' objects}
      495    0.002    0.000    0.003    0.000 tempfile.py:273(rng)
     5977    0.003    0.000    0.003    0.000 {built-in method builtins.__import__}
      495    0.001    0.000    0.003    0.000 carddef.py:133(get_api_url)
      495    0.002    0.000    0.003    0.000 weakref.py:106(remove)
      495    0.002    0.000    0.003    0.000 trans_real.py:101(get)
      495    0.003    0.000    0.003    0.000 formdata.py:1623(__getattr__)
      495    0.001    0.000    0.003    0.000 tempfile.py:245(_sanitize_params)
     1980    0.002    0.000    0.003    0.000 <string>:1(<lambda>)
      495    0.002    0.000    0.002    0.000 afterjobs.py:131(__getstate__)
     5445    0.002    0.000    0.002    0.000 carddata.py:25(get_formdef)
      990    0.001    0.000    0.002    0.000 misc.py:606(default)
    10923    0.002    0.000    0.002    0.000 {method 'add' of 'set' objects}
     8457    0.002    0.000    0.002    0.000 {method 'append' of 'list' objects}
     1983    0.002    0.000    0.002    0.000 {method 'join' of 'str' objects}
     2475    0.002    0.000    0.002    0.000 {method 'split' of 'str' objects}
     3700    0.002    0.000    0.002    0.000 sql.py:2542(get_evolution)
     3967    0.001    0.000    0.002    0.000 encoding.py:85(force_bytes)
      495    0.002    0.000    0.002    0.000 locket.py:153(__init__)
      495    0.001    0.000    0.002    0.000 weakref.py:345(__new__)
     3966    0.002    0.000    0.002    0.000 {method 'tobytes' of 'memoryview' objects}
      990    0.002    0.000    0.002    0.000 formdata.py:695(<listcomp>)
     1486    0.002    0.000    0.002    0.000 workflows.py:690(get_backoffice_fields)
      991    0.001    0.000    0.002    0.000 storage.py:73(fix_key)
      495    0.002    0.000    0.002    0.000 weakref.py:350(__init__)
     2476    0.002    0.000    0.002    0.000 formdef.py:641(get_workflow)
        1    0.000    0.000    0.002    0.002 storage.py:574(get)
        1    0.000    0.000    0.002    0.002 storage.py:682(get_filename)
      495    0.001    0.000    0.002    0.000 posixpath.py:140(basename)
     2475    0.002    0.000    0.002    0.000 {built-in method __new__ of type object at 0x906da0}
     4952    0.001    0.000    0.001    0.000 {built-in method posix.fspath}
     6868    0.001    0.000    0.001    0.000 {method 'getrandbits' of '_random.Random' objects}
      495    0.001    0.000    0.001    0.000 tempfile.py:224(_infer_return_type)
     1980    0.001    0.000    0.001    0.000 {method 'match' of 're.Pattern' objects}
      495    0.001    0.000    0.001    0.000 {built-in method posix.getpid}
      495    0.001    0.000    0.001    0.000 carddata.py:59(get_display_label)
      495    0.001    0.000    0.001    0.000 locket.py:137(acquire)
      990    0.001    0.000    0.001    0.000 sql.py:468(<listcomp>)
     6930    0.001    0.000    0.001    0.000 parse.py:100(_noop)
      496    0.001    0.000    0.001    0.000 formdef.py:579(get_all_fields)
     6437    0.001    0.000    0.001    0.000 {built-in method builtins.len}
      495    0.001    0.000    0.001    0.000 locket.py:148(release)
      990    0.001    0.000    0.001    0.000 formdata.py:816(get_display_id)
      990    0.001    0.000    0.001    0.000 encoding.py:51(force_text)
      990    0.001    0.000    0.001    0.000 {method 'isoformat' of 'datetime.datetime' objects}
      990    0.001    0.000    0.001    0.000 {method 'fileno' of '_io.TextIOWrapper' objects}
      990    0.001    0.000    0.001    0.000 {built-in method _thread.allocate_lock}
     2018    0.001    0.000    0.001    0.000 {method 'replace' of 'str' objects}
     2972    0.001    0.000    0.001    0.000 {method 'endswith' of 'str' objects}
        2    0.001    0.000    0.001    0.000 {method 'commit' of 'psycopg2.extensions.connection' objects}
     1486    0.001    0.000    0.001    0.000 storage.py:446(get_table_name)
     3966    0.001    0.000    0.001    0.000 <ipython-input-64-5bad67047637>:1(<lambda>)
      495    0.000    0.000    0.001    0.000 storage.py:443(is_readonly)
      495    0.001    0.000    0.001    0.000 codecs.py:186(__init__)
     3960    0.001    0.000    0.001    0.000 {method 'bit_length' of 'int' objects}
      495    0.001    0.000    0.001    0.000 {method 'write' of '_io.BufferedWriter' objects}
      495    0.001    0.000    0.001    0.000 {built-in method _weakref._remove_dead_weakref}
      990    0.001    0.000    0.001    0.000 {method 'rfind' of 'str' objects}
     4948    0.001    0.000    0.001    0.000 {method 'items' of 'dict' objects}
     1986    0.001    0.000    0.001    0.000 formdata.py:148(__init__)
      495    0.001    0.000    0.001    0.000 locket.py:102(__init__)
        1    0.000    0.000    0.001    0.001 extensions.py:118(getquoted)
      495    0.001    0.000    0.001    0.000 workflows.py:1978(is_visible)
      495    0.000    0.000    0.000    0.000 {method 'fileno' of '_io.BufferedWriter' objects}
        1    0.000    0.000    0.000    0.000 sql.py:2594(<dictcomp>)
     1980    0.000    0.000    0.000    0.000 {method 'end' of 're.Match' objects}
      495    0.000    0.000    0.000    0.000 {method 'copy' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'close' of '_io.BufferedReader' objects}
      495    0.000    0.000    0.000    0.000 tempfile.py:364(_get_candidate_names)
      495    0.000    0.000    0.000    0.000 formdata.py:473(get_parent)
      495    0.000    0.000    0.000    0.000 {method 'rstrip' of 'str' objects}
      495    0.000    0.000    0.000    0.000 {method 'acquire' of '_thread.lock' objects}
        1    0.000    0.000    0.000    0.000 management.py:4287(<listcomp>)
      495    0.000    0.000    0.000    0.000 sql.py:2536(__init__)
        1    0.000    0.000    0.000    0.000 formdef.py:1950(storage_load)
        1    0.000    0.000    0.000    0.000 {method '__exit__' of 'psycopg2.extensions.cursor' objects}
      990    0.000    0.000    0.000    0.000 {built-in method builtins.issubclass}
        1    0.000    0.000    0.000    0.000 extensions.py:121(<listcomp>)
      495    0.000    0.000    0.000    0.000 {built-in method sys.audit}
      495    0.000    0.000    0.000    0.000 {method '__exit__' of '_thread.lock' objects}
      498    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}
      495    0.000    0.000    0.000    0.000 {method 'release' of '_thread.lock' objects}
        1    0.000    0.000    0.000    0.000 extensions.py:126(<listcomp>)
        1    0.000    0.000    0.000    0.000 {method 'close' of 'psycopg2.extensions.cursor' objects}
      495    0.000    0.000    0.000    0.000 formdata.py:1236(set_last_update_time)
      495    0.000    0.000    0.000    0.000 {built-in method psycopg2._psycopg.adapt}
        1    0.000    0.000    0.000    0.000 {built-in method _pickle.load}
        1    0.000    0.000    0.000    0.000 sql.py:2890(get_data_fields)
      497    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
      495    0.000    0.000    0.000    0.000 formdata.py:1167(workflow_merged_roles_dict)
        1    0.000    0.000    0.000    0.000 formdef.py:218(migrate)
        1    0.000    0.000    0.000    0.000 management.py:4266(__init__)
      495    0.000    0.000    0.000    0.000 {method 'getquoted' of 'psycopg2.extensions.Int' objects}
        1    0.000    0.000    0.000    0.000 storage.py:674(storage_load)
        1    0.000    0.000    0.000    0.000 management.py:4142(__init__)
       67    0.000    0.000    0.000    0.000 utf_8.py:15(decode)
        2    0.000    0.000    0.000    0.000 sql.py:432(parse_clause)
        1    0.000    0.000    0.000    0.000 afterjobs.py:57(__init__)
        2    0.000    0.000    0.000    0.000 sql.py:506(get_connection_and_cursor)
       14    0.000    0.000    0.000    0.000 fields.py:598(migrate)
       34    0.000    0.000    0.000    0.000 sql.py:428(get_field_id)
        1    0.000    0.000    0.000    0.000 uuid.py:713(uuid4)
       67    0.000    0.000    0.000    0.000 {built-in method _codecs.utf_8_decode}
        2    0.000    0.000    0.000    0.000 carddef.py:54(data_class)
        8    0.000    0.000    0.000    0.000 fields.py:1294(migrate)
        2    0.000    0.000    0.000    0.000 sql.py:476(get_connection)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.any}
        1    0.000    0.000    0.000    0.000 uuid.py:138(__init__)
        2    0.000    0.000    0.000    0.000 {method 'cursor' of 'psycopg2.extensions.connection' objects}
        2    0.000    0.000    0.000    0.000 sql.py:204(as_sql)
        4    0.000    0.000    0.000    0.000 fields.py:2126(migrate)
        1    0.000    0.000    0.000    0.000 fields.py:2937(migrate)
       29    0.000    0.000    0.000    0.000 storage.py:713(<genexpr>)
        1    0.000    0.000    0.000    0.000 sql.py:2238(<listcomp>)
        4    0.000    0.000    0.000    0.000 formdef.py:342(data_class_name)
        2    0.000    0.000    0.000    0.000 sql.py:181(as_sql_param)
        1    0.000    0.000    0.000    0.000 sql.py:2972(get_order_by_clause)
        3    0.000    0.000    0.000    0.000 sql.py:114(__init__)
        2    0.000    0.000    0.000    0.000 sql.py:119(as_sql)
        1    0.000    0.000    0.000    0.000 {method 'join' of 'bytes' objects}
        1    0.000    0.000    0.000    0.000 uuid.py:279(__str__)
        4    0.000    0.000    0.000    0.000 {method 'title' of 'str' objects}
       34    0.000    0.000    0.000    0.000 {method 'lower' of 'str' objects}
        1    0.000    0.000    0.000    0.000 formdef.py:1942(__setstate__)
        1    0.000    0.000    0.000    0.000 {built-in method posix.urandom}
        1    0.000    0.000    0.000    0.000 sql.py:2182(get_objects)
        1    0.000    0.000    0.000    0.000 encoder.py:104(__init__)
        1    0.000    0.000    0.000    0.000 extensions.py:111(__init__)
        4    0.000    0.000    0.000    0.000 {built-in method builtins.id}
        1    0.000    0.000    0.000    0.000 {built-in method from_bytes}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 storage.py:440(__init__)
        1    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'count' of 'list' objects}
        2    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
        4    0.000    0.000    0.000    0.000 {built-in method builtins.callable}
        2    0.000    0.000    0.000    0.000 {built-in method builtins.globals}
        1    0.000    0.000    0.000    0.000 {built-in method time.time}
        1    0.000    0.000    0.000    0.000 sql.py:2189(get_order_by_clause)
        1    0.000    0.000    0.000    0.000 extensions.py:115(prepare)
        1    0.000    0.000    0.000    0.000 __init__.py:62(N_)
        1    0.000    0.000    0.000    0.000 management.py:4284(<dictcomp>)
        1    0.000    0.000    0.000    0.000 sql.py:2892(<listcomp>)
        1    0.000    0.000    0.000    0.000 sql.py:2107(get_ids)
        1    0.000    0.000    0.000    0.000 {method 'values' of 'dict' objects}

#3

Updated by Benjamin Dauvergne 3 months ago

Benjamin Dauvergne a écrit :

En fait on perd du temps à écrire des fichiers sur le disque dans AfterJob.increment_count()
[...]

J'ai écrit une autre fonction qui ne passe pas par JsonFileExportAfterJob et là le gros du temps est perdu dans deep_bytes2str et preprocess_struct_time (genre 400ms sur 800), c'est pour après.

PS: sans passer par l'AfterJob, en enlevant evolution et rôles :

In [97]: def f():
    ...:     items = list(carddef.data_class().select([Equal('fbod24ee8d0-8da6-4562-a8e4-f0eca4e0bd26', False)]))
    ...:     carddef.data_class().load_all_evolutions(items)
    ...:     prefetched_users = {str(user.id): user for user in User.select()}
    ...:     prefetched_roles = {str(role.id): role for role in Role.select()}
    ...:     return len(json.dumps([x.get_json_export_dict(anonymise=False, digest_key='default', prefetched_users=users, prefetched_roles=roles, include_evolution=False, include_roles=False, include_files=False
    ...: , include_unnamed_fields=False) for x in items], cls=JSONEncoder))
    ...: 
    ...: 

In [98]: cProfile.run('f()',  sort=SortKey.CUMULATIVE)
         1313426 function calls (1127044 primitive calls) in 1.284 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.284    1.284 {built-in method builtins.exec}
        1    0.000    0.000    1.284    1.284 <string>:1(<module>)
        1    0.001    0.001    1.284    1.284 <ipython-input-97-b42d54d58c4e>:1(f)
3703/3702    0.002    0.000    1.006    0.000 sql.py:629(f)
        1    0.007    0.007    0.840    0.840 sql.py:2585(load_all_evolutions)
     3966    0.021    0.000    0.813    0.000 sql.py:105(pickle_loads)
     1986    0.009    0.000    0.786    0.000 sql.py:2568(_row2evo)
     4052    0.557    0.000    0.577    0.000 {method 'load' of '_pickle.Unpickler' objects}
67813/3966    0.131    0.000    0.211    0.000 storage.py:109(deep_bytes2str)
10092/1986    0.012    0.000    0.179    0.000 storage.py:123(<listcomp>)
        1    0.000    0.000    0.163    0.163 sql.py:2294(select)
      496    0.000    0.000    0.162    0.000 sql.py:2227(select_iterator)
        1    0.000    0.000    0.150    0.150 __init__.py:183(dumps)
        1    0.002    0.002    0.150    0.150 misc.py:603(encode)
      496    0.000    0.000    0.134    0.000 sql.py:2243(retrieve)
      496    0.001    0.000    0.133    0.000 sql.py:2174(get_objects_iterator)
        1    0.001    0.001    0.119    0.119 <ipython-input-97-b42d54d58c4e>:6(<listcomp>)
      495    0.010    0.000    0.118    0.000 formdata.py:1342(get_json_export_dict)
 112944/1    0.083    0.000    0.117    0.117 misc.py:586(preprocess_struct_time)
        1    0.000    0.000    0.117    0.117 misc.py:593(<listcomp>)
      495    0.009    0.000    0.102    0.000 sql.py:2863(_row2ob)
      495    0.022    0.000    0.081    0.000 sql.py:2392(_row2obdata)
   594264    0.071    0.000    0.071    0.000 {built-in method builtins.isinstance}
        2    0.063    0.032    0.064    0.032 {method 'execute' of 'psycopg2.extensions.cursor' objects}
     2483    0.019    0.000    0.039    0.000 {method 'fetchone' of 'psycopg2.extensions.cursor' objects}
      990    0.002    0.000    0.035    0.000 formdata.py:804(get_url)
      990    0.002    0.000    0.033    0.000 carddef.py:121(get_url)
        1    0.000    0.000    0.031    0.031 encoder.py:182(encode)
      990    0.001    0.000    0.031    0.000 publisher.py:164(get_backoffice_url)
        1    0.028    0.028    0.030    0.030 encoder.py:204(iterencode)
      990    0.006    0.000    0.027    0.000 parse.py:516(urljoin)
      495    0.002    0.000    0.027    0.000 formdata.py:1262(get_display_name)
      990    0.001    0.000    0.027    0.000 formdata.py:1332(get_json_dict)
      990    0.019    0.000    0.025    0.000 formdata.py:1287(get_json_data_dict)
     6062    0.012    0.000    0.020    0.000 publisher.py:50(find_class)
      990    0.001    0.000    0.020    0.000 _json.py:164(typecast_json)
      990    0.002    0.000    0.019    0.000 __init__.py:299(loads)
    37519    0.014    0.000    0.019    0.000 encoding.py:51(force_text)
      495    0.000    0.000    0.018    0.000 functional.py:202(__wrapper__)
      495    0.001    0.000    0.017    0.000 functional.py:109(__init__)
39176/37691    0.011    0.000    0.017    0.000 sql.py:466(str_encode)
      990    0.003    0.000    0.017    0.000 decoder.py:332(decode)
    89735    0.016    0.000    0.017    0.000 {built-in method builtins.hasattr}
      495    0.000    0.000    0.017    0.000 formdata.py:807(get_backoffice_url)
      495    0.008    0.000    0.016    0.000 functional.py:125(__prepare_class__)
     1980    0.004    0.000    0.013    0.000 parse.py:369(urlparse)
      990    0.012    0.000    0.012    0.000 decoder.py:343(raw_decode)
        2    0.000    0.000    0.011    0.005 storage.py:511(select)
    13902    0.005    0.000    0.010    0.000 sql.py:2571(<genexpr>)
    45697    0.009    0.000    0.009    0.000 {method 'get' of 'dict' objects}
    52498    0.008    0.000    0.008    0.000 {built-in method builtins.id}
       88    0.000    0.000    0.007    0.000 storage.py:526(<genexpr>)
     6930    0.006    0.000    0.007    0.000 parse.py:111(_coerce_args)
       86    0.000    0.000    0.007    0.000 storage.py:574(get)
      495    0.001    0.000    0.007    0.000 functional.py:190(__mod__)
    28402    0.006    0.000    0.006    0.000 {method 'startswith' of 'str' objects}
       86    0.000    0.000    0.006    0.000 storage.py:682(get_filename)
      990    0.001    0.000    0.006    0.000 parse.py:486(urlunparse)
      495    0.001    0.000    0.006    0.000 functional.py:155(__text_cast)
      495    0.002    0.000    0.006    0.000 formdata.py:1226(get_last_update_time)
      495    0.001    0.000    0.005    0.000 __init__.py:41(gettext)
     2611    0.005    0.000    0.005    0.000 {method 'timetuple' of 'datetime.datetime' objects}
      495    0.002    0.000    0.005    0.000 formdata.py:708(get_visible_status)
     1980    0.003    0.000    0.005    0.000 parse.py:434(urlsplit)
    37519    0.005    0.000    0.005    0.000 {built-in method builtins.issubclass}
      990    0.003    0.000    0.005    0.000 formdata.py:685(get_status)
     6062    0.004    0.000    0.004    0.000 {built-in method builtins.__import__}
     1485    0.001    0.000    0.004    0.000 publisher.py:148(get_frontoffice_url)
    11385    0.003    0.000    0.004    0.000 {built-in method builtins.setattr}
      495    0.000    0.000    0.003    0.000 __init__.py:78(gettext)
    12357    0.003    0.000    0.003    0.000 {built-in method builtins.getattr}
        2    0.000    0.000    0.003    0.002 storage.py:454(keys)
      495    0.001    0.000    0.003    0.000 formdata.py:810(get_api_url)
      990    0.002    0.000    0.003    0.000 parse.py:497(urlunsplit)
      495    0.001    0.000    0.003    0.000 trans_real.py:343(gettext)
        2    0.000    0.000    0.003    0.001 genericpath.py:16(exists)
        2    0.003    0.001    0.003    0.001 {built-in method posix.stat}
       86    0.003    0.000    0.003    0.000 {built-in method io.open}
    15030    0.003    0.000    0.003    0.000 {method 'items' of 'dict' objects}
     4052    0.002    0.000    0.002    0.000 encoding.py:85(force_bytes)
      495    0.001    0.000    0.002    0.000 carddef.py:133(get_api_url)
      990    0.001    0.000    0.002    0.000 misc.py:606(default)
     1485    0.001    0.000    0.002    0.000 publisher.py:1012(get_cfg)
     3816    0.001    0.000    0.002    0.000 monkeypatch.py:46(get_publisher)
     3966    0.002    0.000    0.002    0.000 {method 'tobytes' of 'memoryview' objects}
       86    0.000    0.000    0.002    0.000 storage.py:674(storage_load)
     1980    0.002    0.000    0.002    0.000 {method 'match' of 're.Pattern' objects}
    10923    0.002    0.000    0.002    0.000 {method 'add' of 'set' objects}
     1980    0.001    0.000    0.001    0.000 <string>:1(<lambda>)
     5445    0.001    0.000    0.001    0.000 carddata.py:25(get_formdef)
      495    0.001    0.000    0.001    0.000 gettext.py:496(gettext)
      990    0.001    0.000    0.001    0.000 sql.py:468(<listcomp>)
      496    0.001    0.000    0.001    0.000 formdef.py:579(get_all_fields)
      993    0.001    0.000    0.001    0.000 {method 'join' of 'str' objects}
     3700    0.001    0.000    0.001    0.000 sql.py:2542(get_evolution)
      990    0.001    0.000    0.001    0.000 {method 'isoformat' of 'datetime.datetime' objects}
     6930    0.001    0.000    0.001    0.000 parse.py:100(_noop)
      495    0.001    0.000    0.001    0.000 formdata.py:1623(__getattr__)
      495    0.001    0.000    0.001    0.000 trans_real.py:101(get)
     1986    0.001    0.000    0.001    0.000 formdata.py:148(__init__)
     2476    0.001    0.000    0.001    0.000 formdef.py:641(get_workflow)
     1486    0.001    0.000    0.001    0.000 workflows.py:690(get_backoffice_fields)
        2    0.001    0.000    0.001    0.000 {method 'commit' of 'psycopg2.extensions.connection' objects}
     4002    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}
      990    0.001    0.000    0.001    0.000 formdata.py:695(<listcomp>)
      495    0.000    0.000    0.001    0.000 carddata.py:59(get_display_label)
        1    0.000    0.000    0.001    0.001 extensions.py:118(getquoted)
      175    0.000    0.000    0.001    0.000 posixpath.py:71(join)
     1980    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}
       89    0.000    0.000    0.001    0.000 storage.py:450(get_objects_dir)
     1980    0.001    0.000    0.001    0.000 {built-in method __new__ of type object at 0x906da0}
      990    0.000    0.000    0.000    0.000 formdata.py:816(get_display_id)
       86    0.000    0.000    0.000    0.000 {built-in method builtins.any}
        1    0.000    0.000    0.000    0.000 sql.py:2594(<dictcomp>)
     1981    0.000    0.000    0.000    0.000 {built-in method builtins.len}
     1980    0.000    0.000    0.000    0.000 {method 'end' of 're.Match' objects}
      495    0.000    0.000    0.000    0.000 sql.py:2536(__init__)
       86    0.000    0.000    0.000    0.000 {method 'close' of '_io.BufferedReader' objects}
      688    0.000    0.000    0.000    0.000 storage.py:713(<genexpr>)
     1199    0.000    0.000    0.000    0.000 {method 'replace' of 'str' objects}
        1    0.000    0.000    0.000    0.000 extensions.py:121(<listcomp>)
        1    0.000    0.000    0.000    0.000 {method '__exit__' of 'psycopg2.extensions.cursor' objects}
        1    0.000    0.000    0.000    0.000 {built-in method posix.listdir}
        1    0.000    0.000    0.000    0.000 extensions.py:126(<listcomp>)
      495    0.000    0.000    0.000    0.000 {method 'mro' of 'type' objects}
        1    0.000    0.000    0.000    0.000 {method 'close' of 'psycopg2.extensions.cursor' objects}
      495    0.000    0.000    0.000    0.000 workflows.py:1978(is_visible)
      495    0.000    0.000    0.000    0.000 formdata.py:1236(set_last_update_time)
      495    0.000    0.000    0.000    0.000 {built-in method psycopg2._psycopg.adapt}
      495    0.000    0.000    0.000    0.000 formdata.py:473(get_parent)
        1    0.000    0.000    0.000    0.000 sql.py:2890(get_data_fields)
      495    0.000    0.000    0.000    0.000 formdata.py:1167(workflow_merged_roles_dict)
      172    0.000    0.000    0.000    0.000 storage.py:73(fix_key)
      497    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
      495    0.000    0.000    0.000    0.000 {method 'getquoted' of 'psycopg2.extensions.Int' objects}
        2    0.000    0.000    0.000    0.000 sql.py:506(get_connection_and_cursor)
      175    0.000    0.000    0.000    0.000 posixpath.py:41(_get_sep)
        2    0.000    0.000    0.000    0.000 sql.py:432(parse_clause)
       67    0.000    0.000    0.000    0.000 utf_8.py:15(decode)
        1    0.000    0.000    0.000    0.000 storage.py:460(<listcomp>)
        2    0.000    0.000    0.000    0.000 {method 'cursor' of 'psycopg2.extensions.connection' objects}
        2    0.000    0.000    0.000    0.000 carddef.py:54(data_class)
       34    0.000    0.000    0.000    0.000 sql.py:428(get_field_id)
        1    0.000    0.000    0.000    0.000 <ipython-input-97-b42d54d58c4e>:5(<dictcomp>)
      175    0.000    0.000    0.000    0.000 {method 'endswith' of 'str' objects}
       86    0.000    0.000    0.000    0.000 roles.py:53(migrate)
       86    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
      175    0.000    0.000    0.000    0.000 {built-in method posix.fspath}
       67    0.000    0.000    0.000    0.000 {built-in method _codecs.utf_8_decode}
       89    0.000    0.000    0.000    0.000 storage.py:446(get_table_name)
        2    0.000    0.000    0.000    0.000 sql.py:181(as_sql_param)
        4    0.000    0.000    0.000    0.000 formdef.py:342(data_class_name)
        2    0.000    0.000    0.000    0.000 sql.py:476(get_connection)
        2    0.000    0.000    0.000    0.000 sql.py:204(as_sql)
        1    0.000    0.000    0.000    0.000 {method 'join' of 'bytes' objects}
        3    0.000    0.000    0.000    0.000 sql.py:114(__init__)
        2    0.000    0.000    0.000    0.000 sql.py:119(as_sql)
        1    0.000    0.000    0.000    0.000 sql.py:2238(<listcomp>)
        4    0.000    0.000    0.000    0.000 {method 'title' of 'str' objects}
       34    0.000    0.000    0.000    0.000 {method 'lower' of 'str' objects}
        1    0.000    0.000    0.000    0.000 extensions.py:111(__init__)
        1    0.000    0.000    0.000    0.000 sql.py:2972(get_order_by_clause)
        1    0.000    0.000    0.000    0.000 sql.py:2182(get_objects)
        1    0.000    0.000    0.000    0.000 encoder.py:104(__init__)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 <ipython-input-97-b42d54d58c4e>:4(<dictcomp>)
        4    0.000    0.000    0.000    0.000 {built-in method builtins.callable}
        2    0.000    0.000    0.000    0.000 storage.py:481(sort_results)
        2    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 extensions.py:115(prepare)
        2    0.000    0.000    0.000    0.000 {built-in method builtins.globals}
        1    0.000    0.000    0.000    0.000 sql.py:2892(<listcomp>)
        1    0.000    0.000    0.000    0.000 sql.py:2189(get_order_by_clause)

#4

Updated by Lauréline Guérin 3 months ago

mais pourquoi loader les evolutions si on n'en veut pas dans le résultat ?

#5

Updated by Benjamin Dauvergne 3 months ago

Bizarre que ce soit la seule chose qui te saute aux yeux, mais bon il y a une réponse, parce que même si on en veut pas ça les charge quand même :

In [22]: def f():
    ...:     items = list(carddef.data_class().select([Equal('fbod24ee8d0-8da6-4562-a8e4-f0eca4e0bd26', False)]))
    ...:     users = {str(user.id): user for user in User.select()}
    ...:     roles = {str(role.id): role for role in Role.select()}
    ...:     return len(json.dumps([x.get_json_export_dict(anonymise=False, digest_key='default', prefetched_users=users, prefetched_roles=roles, include_evolution=False, include_roles=False, include_files=False
    ...: , include_unnamed_fields=False) for x in items], cls=JSONEncoder))
    ...: 

In [23]: cProfile.run('f()',  sort=SortKey.CUMULATIVE)
         1324320 function calls (1137854 primitive calls) in 1.355 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    1.355    1.355 {built-in method builtins.exec}
        1    0.000    0.000    1.355    1.355 <string>:1(<module>)
        1    0.002    0.002    1.355    1.355 <ipython-input-22-6e5cfbf65bea>:1(f)
        1    0.001    0.001    1.029    1.029 <ipython-input-22-6e5cfbf65bea>:5(<listcomp>)
      495    0.017    0.000    1.028    0.002 formdata.py:1342(get_json_export_dict)
3702/3701    0.004    0.000    1.014    0.000 sql.py:629(f)
      495    0.003    0.000    0.859    0.002 formdata.py:1226(get_last_update_time)
     3700    0.012    0.000    0.850    0.000 sql.py:2542(get_evolution)
     3966    0.031    0.000    0.382    0.000 sql.py:105(pickle_loads)
     1986    0.018    0.000    0.372    0.000 sql.py:2568(_row2evo)
      496    0.362    0.001    0.368    0.001 {method 'execute' of 'psycopg2.extensions.cursor' objects}
67885/3966    0.147    0.000    0.232    0.000 storage.py:109(deep_bytes2str)
10104/1986    0.012    0.000    0.199    0.000 storage.py:123(<listcomp>)
        1    0.000    0.000    0.160    0.160 sql.py:2294(select)
      496    0.000    0.000    0.160    0.000 sql.py:2227(select_iterator)
        1    0.000    0.000    0.155    0.155 __init__.py:183(dumps)
        1    0.002    0.002    0.155    0.155 misc.py:603(encode)
      496    0.000    0.000    0.129    0.000 sql.py:2243(retrieve)
      496    0.001    0.000    0.129    0.000 sql.py:2174(get_objects_iterator)
 112944/1    0.084    0.000    0.119    0.119 misc.py:586(preprocess_struct_time)
        1    0.000    0.000    0.119    0.119 misc.py:593(<listcomp>)
     4052    0.087    0.000    0.115    0.000 {method 'load' of '_pickle.Unpickler' objects}
      496    0.107    0.000    0.107    0.000 {method 'commit' of 'psycopg2.extensions.connection' objects}
      495    0.009    0.000    0.101    0.000 sql.py:2863(_row2ob)
      495    0.022    0.000    0.080    0.000 sql.py:2392(_row2obdata)
   594504    0.074    0.000    0.074    0.000 {built-in method builtins.isinstance}
      990    0.003    0.000    0.055    0.000 formdata.py:804(get_url)
      990    0.003    0.000    0.051    0.000 carddef.py:121(get_url)
      990    0.003    0.000    0.046    0.000 publisher.py:164(get_backoffice_url)
     2977    0.021    0.000    0.040    0.000 {method 'fetchone' of 'psycopg2.extensions.cursor' objects}
      990    0.009    0.000    0.039    0.000 parse.py:516(urljoin)
      495    0.003    0.000    0.039    0.000 formdata.py:1262(get_display_name)
      990    0.002    0.000    0.038    0.000 formdata.py:1332(get_json_dict)
      990    0.028    0.000    0.036    0.000 formdata.py:1287(get_json_data_dict)
        1    0.000    0.000    0.034    0.034 encoder.py:182(encode)
        1    0.029    0.029    0.032    0.032 encoder.py:204(iterencode)
     6062    0.017    0.000    0.028    0.000 publisher.py:50(find_class)
      495    0.001    0.000    0.024    0.000 functional.py:202(__wrapper__)
      495    0.001    0.000    0.023    0.000 functional.py:109(__init__)
    89275    0.021    0.000    0.023    0.000 {built-in method builtins.hasattr}
      495    0.010    0.000    0.022    0.000 functional.py:125(__prepare_class__)
      495    0.001    0.000    0.021    0.000 formdata.py:807(get_backoffice_url)
    37555    0.015    0.000    0.020    0.000 encoding.py:51(force_text)
     1980    0.007    0.000    0.019    0.000 parse.py:369(urlparse)
39176/37691    0.012    0.000    0.019    0.000 sql.py:466(str_encode)
      990    0.001    0.000    0.018    0.000 _json.py:164(typecast_json)
      990    0.001    0.000    0.017    0.000 __init__.py:299(loads)
      990    0.003    0.000    0.015    0.000 decoder.py:332(decode)
    13902    0.006    0.000    0.013    0.000 sql.py:2571(<genexpr>)
      495    0.002    0.000    0.012    0.000 functional.py:190(__mod__)
    43711    0.012    0.000    0.012    0.000 {method 'get' of 'dict' objects}
      990    0.011    0.000    0.011    0.000 decoder.py:343(raw_decode)
      495    0.001    0.000    0.010    0.000 functional.py:155(__text_cast)
     6930    0.008    0.000    0.010    0.000 parse.py:111(_coerce_args)
      495    0.002    0.000    0.009    0.000 __init__.py:41(gettext)
        2    0.000    0.000    0.009    0.004 storage.py:511(select)
     2611    0.009    0.000    0.009    0.000 {method 'timetuple' of 'datetime.datetime' objects}
      495    0.003    0.000    0.009    0.000 formdata.py:708(get_visible_status)
      990    0.002    0.000    0.008    0.000 parse.py:486(urlunparse)
       88    0.000    0.000    0.008    0.000 storage.py:526(<genexpr>)
    52570    0.008    0.000    0.008    0.000 {built-in method builtins.id}
       86    0.000    0.000    0.008    0.000 storage.py:574(get)
     1980    0.004    0.000    0.007    0.000 parse.py:434(urlsplit)
    28438    0.007    0.000    0.007    0.000 {method 'startswith' of 'str' objects}
      990    0.004    0.000    0.007    0.000 formdata.py:685(get_status)
       86    0.001    0.000    0.007    0.000 storage.py:682(get_filename)
      496    0.001    0.000    0.006    0.000 sql.py:506(get_connection_and_cursor)
     4018    0.003    0.000    0.006    0.000 utf_8.py:15(decode)
      495    0.001    0.000    0.006    0.000 __init__.py:78(gettext)
    13838    0.006    0.000    0.006    0.000 {built-in method builtins.getattr}
     6062    0.006    0.000    0.006    0.000 {built-in method builtins.__import__}
     1485    0.002    0.000    0.006    0.000 publisher.py:148(get_frontoffice_url)
    37555    0.005    0.000    0.005    0.000 {built-in method builtins.issubclass}
      495    0.002    0.000    0.005    0.000 trans_real.py:343(gettext)
      495    0.001    0.000    0.005    0.000 formdata.py:810(get_api_url)
     4804    0.002    0.000    0.004    0.000 monkeypatch.py:46(get_publisher)
      990    0.002    0.000    0.004    0.000 parse.py:497(urlunsplit)
    15042    0.004    0.000    0.004    0.000 {method 'items' of 'dict' objects}
     1485    0.002    0.000    0.003    0.000 publisher.py:1012(get_cfg)
       86    0.003    0.000    0.003    0.000 {built-in method io.open}
    11385    0.003    0.000    0.003    0.000 {built-in method builtins.setattr}
      495    0.001    0.000    0.003    0.000 carddef.py:133(get_api_url)
      496    0.003    0.000    0.003    0.000 {method 'cursor' of 'psycopg2.extensions.connection' objects}
     4052    0.002    0.000    0.003    0.000 encoding.py:85(force_bytes)
     4018    0.003    0.000    0.003    0.000 {built-in method _codecs.utf_8_decode}
      993    0.002    0.000    0.002    0.000 {method 'join' of 'str' objects}
      990    0.001    0.000    0.002    0.000 misc.py:606(default)
      495    0.001    0.000    0.002    0.000 gettext.py:496(gettext)
     3966    0.002    0.000    0.002    0.000 {method 'tobytes' of 'memoryview' objects}
     1980    0.001    0.000    0.002    0.000 <string>:1(<lambda>)
     5445    0.002    0.000    0.002    0.000 carddata.py:25(get_formdef)
    10923    0.002    0.000    0.002    0.000 {method 'add' of 'set' objects}
       86    0.000    0.000    0.002    0.000 storage.py:674(storage_load)
      496    0.001    0.000    0.002    0.000 sql.py:476(get_connection)
      495    0.002    0.000    0.002    0.000 formdata.py:1623(__getattr__)
      495    0.001    0.000    0.001    0.000 trans_real.py:101(get)
     1980    0.001    0.000    0.001    0.000 {method 'match' of 're.Pattern' objects}
      496    0.001    0.000    0.001    0.000 formdef.py:579(get_all_fields)
     1986    0.001    0.000    0.001    0.000 formdata.py:148(__init__)
      990    0.001    0.000    0.001    0.000 sql.py:468(<listcomp>)
      990    0.001    0.000    0.001    0.000 formdata.py:695(<listcomp>)
     6930    0.001    0.000    0.001    0.000 parse.py:100(_noop)
      990    0.001    0.000    0.001    0.000 {method 'isoformat' of 'datetime.datetime' objects}
     1486    0.001    0.000    0.001    0.000 workflows.py:690(get_backoffice_fields)
     2476    0.001    0.000    0.001    0.000 formdef.py:641(get_workflow)
     4002    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}
     1980    0.001    0.000    0.001    0.000 {built-in method __new__ of type object at 0x906da0}
     1980    0.001    0.000    0.001    0.000 {method 'split' of 'str' objects}
      990    0.001    0.000    0.001    0.000 formdata.py:816(get_display_id)
        2    0.000    0.000    0.001    0.000 storage.py:454(keys)
      495    0.000    0.000    0.001    0.000 carddata.py:59(get_display_label)
      495    0.001    0.000    0.001    0.000 {method 'close' of 'psycopg2.extensions.cursor' objects}
      175    0.000    0.000    0.001    0.000 posixpath.py:71(join)
       89    0.000    0.000    0.001    0.000 storage.py:450(get_objects_dir)
        1    0.001    0.001    0.001    0.001 {built-in method posix.listdir}
       86    0.000    0.000    0.000    0.000 {built-in method builtins.any}
     1981    0.000    0.000    0.000    0.000 {built-in method builtins.len}
     1199    0.000    0.000    0.000    0.000 {method 'replace' of 'str' objects}
      495    0.000    0.000    0.000    0.000 {method 'mro' of 'type' objects}
      495    0.000    0.000    0.000    0.000 workflows.py:1978(is_visible)
     1980    0.000    0.000    0.000    0.000 {method 'end' of 're.Match' objects}
       86    0.000    0.000    0.000    0.000 {method 'close' of '_io.BufferedReader' objects}
      495    0.000    0.000    0.000    0.000 formdata.py:473(get_parent)
      688    0.000    0.000    0.000    0.000 storage.py:713(<genexpr>)
      495    0.000    0.000    0.000    0.000 sql.py:2536(__init__)
        1    0.000    0.000    0.000    0.000 {method '__exit__' of 'psycopg2.extensions.cursor' objects}
      495    0.000    0.000    0.000    0.000 formdata.py:1236(set_last_update_time)
        1    0.000    0.000    0.000    0.000 sql.py:2890(get_data_fields)
      172    0.000    0.000    0.000    0.000 storage.py:73(fix_key)
      495    0.000    0.000    0.000    0.000 formdata.py:1167(workflow_merged_roles_dict)
      496    0.000    0.000    0.000    0.000 {method 'keys' of 'dict' objects}
      175    0.000    0.000    0.000    0.000 posixpath.py:41(_get_sep)
        2    0.000    0.000    0.000    0.000 genericpath.py:16(exists)
        2    0.000    0.000    0.000    0.000 {built-in method posix.stat}
        1    0.000    0.000    0.000    0.000 storage.py:460(<listcomp>)
        2    0.000    0.000    0.000    0.000 sql.py:432(parse_clause)
       34    0.000    0.000    0.000    0.000 sql.py:428(get_field_id)
      175    0.000    0.000    0.000    0.000 {method 'endswith' of 'str' objects}
        1    0.000    0.000    0.000    0.000 <ipython-input-22-6e5cfbf65bea>:4(<dictcomp>)
       86    0.000    0.000    0.000    0.000 roles.py:53(migrate)
      175    0.000    0.000    0.000    0.000 {built-in method posix.fspath}
       86    0.000    0.000    0.000    0.000 {method 'encode' of 'str' objects}
        1    0.000    0.000    0.000    0.000 carddef.py:54(data_class)
       89    0.000    0.000    0.000    0.000 storage.py:446(get_table_name)
        3    0.000    0.000    0.000    0.000 sql.py:114(__init__)
        2    0.000    0.000    0.000    0.000 sql.py:204(as_sql)
        2    0.000    0.000    0.000    0.000 formdef.py:342(data_class_name)
        2    0.000    0.000    0.000    0.000 sql.py:181(as_sql_param)
        1    0.000    0.000    0.000    0.000 sql.py:2238(<listcomp>)
        2    0.000    0.000    0.000    0.000 sql.py:119(as_sql)
       34    0.000    0.000    0.000    0.000 {method 'lower' of 'str' objects}
        1    0.000    0.000    0.000    0.000 sql.py:2972(get_order_by_clause)
        2    0.000    0.000    0.000    0.000 {method 'title' of 'str' objects}
        1    0.000    0.000    0.000    0.000 sql.py:2182(get_objects)
        1    0.000    0.000    0.000    0.000 encoder.py:104(__init__)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        2    0.000    0.000    0.000    0.000 {method 'update' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 <ipython-input-22-6e5cfbf65bea>:3(<dictcomp>)
        4    0.000    0.000    0.000    0.000 {built-in method builtins.callable}
        2    0.000    0.000    0.000    0.000 storage.py:481(sort_results)
        2    0.000    0.000    0.000    0.000 {built-in method builtins.globals}
        1    0.000    0.000    0.000    0.000 sql.py:2892(<listcomp>)
        1    0.000    0.000    0.000    0.000 sql.py:2189(get_order_by_clause)

3700 appels à get_evolution qui coûte pas beaucoup plus cher que load_all_evolution vu que tout le temps est passé dans pickle&co, mais c'est un artefact de cProfile, en vrai ça va plus vite avec load_all_evolution():
In [24]: %timeit f() # sans load_all_evolution
959 ms ± 21.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
...
In [26]: %timeit f() # avec load_all_evolution
434 ms ± 13.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Sur l'instance Toulouse les temps SQL varient beaucoup mais par contre les temps passés localement dans pickle&co sont à peu près stable sous cProfile, il faudrait placer un breakpoint dans get_evolution() pour voir qui l'appelle, cProfile ne donne pas le callgraph (ou alors je n'ai pas trouvé la bonne option).

#6

Updated by Benjamin Dauvergne 3 months ago

Y a une méthode print_callers() bien utile sur l'objet pstat et donc les coupables sont:
  • cette ligne où il faudrait inverse self.evolution et include_evolution pour bénéficier du court-circuitage :
            if self.evolution and include_evolution:
    
  • get_visible_status() : là ce sera plus difficile de faire sans
  • get_last_update_time() : logiquement on devrait pouvoir lire la valeur depuis la colonne mais le design est de toujours lire la valeur à jour depuis les évolutions, la colonne en ne sert qu'à filtrer, ceci pour que la valeur écrite soit toujours la dernière valeur à jour issue des évolutions (et le setter défini ne semble servir à rien j'ai l'impression)

PS: vu que c'est les pickle_loads qui coûtent cher et qu'ici on ne lit pas evo.parts ça pourrait être utile d'avoir du lazy loading à cet endroit.

#7

Updated by Pierre Ducroquet 3 months ago

Pour le temps perdu sur les écritures disque, ne peut-on pas au moins à court terme tricher et modifier le filename du JsonFileExportAfterJob pour que ce soit /dev/shm/machin.json dans le cas des exports à destination de l'API ? Comme ça on va écrire en RAM, ça sera beaucoup plus rapide (à défaut de supprimer tous ces appels inutiles)
Ou écrire un JsonMemExportAfterJob (ou un truc mieux nommé) qui évite ça... Parce que là c'est quand même triste, surtout pour les cas où on est mal en IOs (HDS...)

#8

Updated by Benjamin Dauvergne 3 months ago

  • Assignee set to Benjamin Dauvergne

Je vais surtout supprimer ce increment_count dans le cas du web service.

#10

Updated by Benjamin Dauvergne 3 months ago

#11

Updated by Benjamin Dauvergne 3 months ago

  • Related to Development #69108: api: performance du endpoint list, permettre d'exclure les evolutions added
#12

Updated by Benjamin Dauvergne 3 months ago

Voilà pour le gros des perfs, j'ai modifié localement sur toulouse-test-x (voir ticket #69080 pour tester) et ça charge le tableau en 2/3 secondes maintenant.

#13

Updated by Lauréline Guérin 3 months ago

  • Status changed from Solution proposée to Solution validée
#14

Updated by Benjamin Dauvergne 3 months ago

  • Blocked by Development #69109: api: performance du endpoint list, charger les parts de manière paresseuse added
#15

Updated by Benjamin Dauvergne 3 months ago

  • Blocked by deleted (Development #69109: api: performance du endpoint list, charger les parts de manière paresseuse)
#16

Updated by Benjamin Dauvergne 3 months ago

  • Related to Development #69109: api: performance du endpoint list, charger les parts de manière paresseuse added
#17

Updated by Benjamin Dauvergne 3 months ago

  • Status changed from Solution validée to Résolu (à déployer)
commit 0296a9e8a368e7336c2f22f1803739d22d68d3d0
Author: Benjamin Dauvergne <bdauvergne@entrouvert.com>
Date:   Thu Sep 15 09:41:56 2022 +0200

    api: does not write to disk during form/card listing (#69090)

    The call to increment_count() is only beneficial to the JSON file export
    case.
#18

Updated by Benjamin Dauvergne 3 months ago

Ça devrait accélérer le chargement des bases bijoe aussi et moins charger le partage NFS pendant ce temps.

#19

Updated by Pierre Ducroquet 3 months ago

Benjamin Dauvergne a écrit :

Ça devrait accélérer le chargement des bases bijoe aussi et moins charger le partage NFS pendant ce temps.

Tout à fait, j'avais déjà vu de la charge sur le NFS que je ne m'expliquais pas. Je verrai après le déploiement ce que ça donne en prod à coup de fatrace...

#20

Updated by Benjamin Dauvergne 3 months ago

  • Related to Development #69126: api: performance, l'appel à preprocess_struct_time dans JSONEncoder.encode est coûteux added
#21

Updated by Transition automatique 3 months ago

  • Status changed from Résolu (à déployer) to Solution déployée
#22

Updated by Transition automatique 19 days ago

Automatic expiration

Also available in: Atom PDF