Development #69090
Requête web beaucoup trop longue sur /api/cards
Début:
14 septembre 2022
Echéance:
% réalisé:
0%
Temps estimé:
Patch proposed:
Oui
Planning:
Non
Description
Observé sur toulouse preprod, #69080 : la liste des 501 éléments prend 10 secondes à être générée.
Plusieurs pistes à suivre:
- réduire le volume du full=on (on passe de 10s à 700ms a priori)
- comprendre où sont passées les 10 secondes
Fichiers
Demandes liées
Révisions associées
Historique
Mis à jour par Benjamin Dauvergne il y a plus d'un an
En fait on perd du temps à écrire des fichiers sur le disque dans AfterJob.increment_count()
In [87]: def f():
...: items = list(carddef.data_class().select([Equal('fbod24ee8d0-8da6-4562-a8e4-f0eca4e0bd26', False)]))
...: carddef.data_class().load_all_evolutions(items)
...: data = JsonFileExportAfterJob(carddef).create_json_export(items, user=None, anonymise=False, digest_key='default', include_evolution=False, include_roles=False, include_files=False, include_unnamed_
...: fields=False)
...: return len(json.dumps(data, cls=JSONEncoder))
...:
In [88]: cProfile.run('f()', sort=SortKey.CUMULATIVE)
817242 function calls (737463 primitive calls) in 9.396 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 9.396 9.396 {built-in method builtins.exec}
1 0.001 0.001 9.396 9.396 <string>:1(<module>)
1 0.000 0.000 9.395 9.395 <ipython-input-87-dd7a1680e2f2>:1(f)
1 0.006 0.006 8.448 8.448 management.py:4270(create_json_export)
495 0.002 0.000 8.190 0.017 afterjobs.py:78(increment_count)
495 0.023 0.000 8.188 0.017 storage.py:830(store)
495 0.004 0.000 6.427 0.013 storage.py:80(atomic_write)
495 0.017 0.000 6.423 0.013 storage.py:84(doit)
495 0.002 0.000 2.785 0.006 tempfile.py:435(mkstemp)
495 0.009 0.000 2.780 0.006 tempfile.py:378(_mkstemp_inner)
495 2.725 0.006 2.725 0.006 {built-in method posix.open}
495 1.456 0.003 1.456 0.003 {built-in method posix.fsync}
495 1.093 0.002 1.093 0.002 {built-in method posix.rename}
495 0.868 0.002 0.868 0.002 {built-in method posix.fchmod}
495 0.854 0.002 0.854 0.002 {built-in method posix.utime}
3704/3703 0.002 0.000 0.843 0.000 sql.py:629(f)
1 0.005 0.005 0.705 0.705 sql.py:2585(load_all_evolutions)
3966 0.016 0.000 0.655 0.000 sql.py:105(pickle_loads)
1986 0.008 0.000 0.653 0.000 sql.py:2568(_row2evo)
3967 0.614 0.000 0.633 0.000 {method 'load' of '_pickle.Unpickler' objects}
495 0.001 0.000 0.369 0.001 locket.py:122(__enter__)
495 0.004 0.000 0.368 0.001 locket.py:105(acquire)
495 0.002 0.000 0.362 0.001 locket.py:160(acquire)
495 0.001 0.000 0.281 0.001 locket.py:126(__exit__)
495 0.003 0.000 0.281 0.001 locket.py:117(release)
495 0.004 0.000 0.276 0.001 locket.py:173(release)
990 0.259 0.000 0.259 0.000 {built-in method fcntl.flock}
495 0.025 0.000 0.249 0.001 formdata.py:1342(get_json_export_dict)
991 0.227 0.000 0.234 0.000 {built-in method io.open}
495 0.168 0.000 0.168 0.000 {method 'close' of '_io.BufferedWriter' objects}
990 0.003 0.000 0.152 0.000 genericpath.py:16(exists)
990 0.149 0.000 0.149 0.000 {built-in method posix.stat}
495 0.146 0.000 0.146 0.000 {method 'close' of '_io.TextIOWrapper' objects}
495 0.002 0.000 0.138 0.000 locket.py:44(_lock_file_blocking)
1 0.000 0.000 0.134 0.134 sql.py:2294(select)
496 0.000 0.000 0.134 0.000 sql.py:2227(select_iterator)
495 0.002 0.000 0.126 0.000 locket.py:59(_unlock_file)
1 0.000 0.000 0.108 0.108 __init__.py:183(dumps)
1 0.001 0.001 0.108 0.108 misc.py:603(encode)
496 0.000 0.000 0.102 0.000 sql.py:2243(retrieve)
496 0.001 0.000 0.102 0.000 sql.py:2174(get_objects_iterator)
78294/1 0.060 0.000 0.084 0.084 misc.py:586(preprocess_struct_time)
1 0.000 0.000 0.084 0.084 misc.py:593(<listcomp>)
495 0.006 0.000 0.078 0.000 formdata.py:1262(get_display_name)
495 0.009 0.000 0.072 0.000 sql.py:2863(_row2ob)
2 0.066 0.033 0.067 0.034 {method 'execute' of 'psycopg2.extensions.cursor' objects}
990 0.003 0.000 0.061 0.000 formdata.py:804(get_url)
990 0.004 0.000 0.057 0.000 carddef.py:121(get_url)
990 0.004 0.000 0.053 0.000 publisher.py:164(get_backoffice_url)
495 0.021 0.000 0.051 0.000 sql.py:2392(_row2obdata)
495 0.002 0.000 0.048 0.000 functional.py:202(__wrapper__)
495 0.002 0.000 0.046 0.000 functional.py:109(__init__)
990 0.002 0.000 0.046 0.000 formdata.py:1332(get_json_dict)
495 0.014 0.000 0.044 0.000 functional.py:125(__prepare_class__)
990 0.010 0.000 0.044 0.000 parse.py:516(urljoin)
990 0.035 0.000 0.044 0.000 formdata.py:1287(get_json_data_dict)
2483 0.018 0.000 0.036 0.000 {method 'fetchone' of 'psycopg2.extensions.cursor' objects}
301842 0.033 0.000 0.033 0.000 {built-in method builtins.isinstance}
495 0.008 0.000 0.029 0.000 locket.py:67(lock_file)
63899 0.023 0.000 0.026 0.000 {built-in method builtins.hasattr}
1 0.000 0.000 0.023 0.023 encoder.py:182(encode)
1 0.020 0.020 0.022 0.022 encoder.py:204(iterencode)
495 0.003 0.000 0.022 0.000 functional.py:190(__mod__)
495 0.003 0.000 0.022 0.000 posixpath.py:372(abspath)
495 0.001 0.000 0.021 0.000 {built-in method builtins.next}
1980 0.008 0.000 0.021 0.000 parse.py:369(urlparse)
495 0.001 0.000 0.021 0.000 storage.py:826(storage_dumps)
495 0.001 0.000 0.021 0.000 formdata.py:807(get_backoffice_url)
495 0.018 0.000 0.020 0.000 {built-in method _pickle.dumps}
495 0.003 0.000 0.020 0.000 tempfile.py:284(__next__)
991 0.006 0.000 0.019 0.000 storage.py:450(get_objects_dir)
495 0.001 0.000 0.019 0.000 functional.py:155(__text_cast)
990 0.001 0.000 0.018 0.000 _json.py:164(typecast_json)
5977 0.012 0.000 0.018 0.000 publisher.py:50(find_class)
495 0.004 0.000 0.018 0.000 __init__.py:41(gettext)
990 0.001 0.000 0.018 0.000 __init__.py:299(loads)
495 0.005 0.000 0.017 0.000 os.py:1019(fdopen)
39176/37691 0.011 0.000 0.017 0.000 sql.py:466(str_encode)
2477 0.011 0.000 0.016 0.000 posixpath.py:71(join)
990 0.002 0.000 0.016 0.000 decoder.py:332(decode)
495 0.002 0.000 0.015 0.000 storage.py:909(update_indexes)
495 0.002 0.000 0.013 0.000 tempfile.py:287(<listcomp>)
495 0.004 0.000 0.013 0.000 formdata.py:1226(get_last_update_time)
495 0.013 0.000 0.013 0.000 {method 'mro' of 'type' objects}
46109 0.012 0.000 0.012 0.000 {method 'get' of 'dict' objects}
495 0.004 0.000 0.012 0.000 formdata.py:708(get_visible_status)
495 0.002 0.000 0.012 0.000 __init__.py:78(gettext)
495 0.012 0.000 0.012 0.000 {method 'flush' of '_io.BufferedWriter' objects}
3960 0.004 0.000 0.011 0.000 random.py:344(choice)
990 0.011 0.000 0.011 0.000 decoder.py:343(raw_decode)
495 0.004 0.000 0.010 0.000 locket.py:76(_create_lock_file)
495 0.004 0.000 0.010 0.000 trans_real.py:343(gettext)
6930 0.008 0.000 0.010 0.000 parse.py:111(_coerce_args)
495 0.006 0.000 0.009 0.000 posixpath.py:334(normpath)
990 0.005 0.000 0.009 0.000 formdata.py:685(get_status)
495 0.002 0.000 0.009 0.000 posixpath.py:60(isabs)
3962 0.007 0.000 0.009 0.000 posixpath.py:41(_get_sep)
13902 0.004 0.000 0.009 0.000 sql.py:2571(<genexpr>)
990 0.002 0.000 0.008 0.000 parse.py:486(urlunparse)
1980 0.004 0.000 0.008 0.000 parse.py:434(urlsplit)
14451 0.007 0.000 0.007 0.000 {built-in method builtins.getattr}
495 0.003 0.000 0.007 0.000 weakref.py:162(__setitem__)
3960 0.005 0.000 0.007 0.000 random.py:238(_randbelow_with_getrandbits)
495 0.003 0.000 0.007 0.000 _bootlocale.py:33(getpreferredencoding)
1485 0.002 0.000 0.006 0.000 publisher.py:148(get_frontoffice_url)
4464 0.003 0.000 0.006 0.000 monkeypatch.py:46(get_publisher)
2611 0.006 0.000 0.006 0.000 {method 'timetuple' of 'datetime.datetime' objects}
495 0.002 0.000 0.005 0.000 formdata.py:810(get_api_url)
495 0.002 0.000 0.005 0.000 gettext.py:496(gettext)
990 0.003 0.000 0.004 0.000 parse.py:497(urlunsplit)
495 0.004 0.000 0.004 0.000 weakref.py:192(get)
495 0.002 0.000 0.004 0.000 posixpath.py:150(dirname)
495 0.004 0.000 0.004 0.000 {built-in method _locale.nl_langinfo}
1485 0.002 0.000 0.004 0.000 publisher.py:1012(get_cfg)
11385 0.003 0.000 0.003 0.000 {built-in method builtins.setattr}
495 0.003 0.000 0.003 0.000 locket.py:131(__init__)
6437 0.003 0.000 0.003 0.000 {method 'startswith' of 'str' objects}
495 0.002 0.000 0.003 0.000 tempfile.py:273(rng)
5977 0.003 0.000 0.003 0.000 {built-in method builtins.__import__}
495 0.001 0.000 0.003 0.000 carddef.py:133(get_api_url)
495 0.002 0.000 0.003 0.000 weakref.py:106(remove)
495 0.002 0.000 0.003 0.000 trans_real.py:101(get)
495 0.003 0.000 0.003 0.000 formdata.py:1623(__getattr__)
495 0.001 0.000 0.003 0.000 tempfile.py:245(_sanitize_params)
1980 0.002 0.000 0.003 0.000 <string>:1(<lambda>)
495 0.002 0.000 0.002 0.000 afterjobs.py:131(__getstate__)
5445 0.002 0.000 0.002 0.000 carddata.py:25(get_formdef)
990 0.001 0.000 0.002 0.000 misc.py:606(default)
10923 0.002 0.000 0.002 0.000 {method 'add' of 'set' objects}
8457 0.002 0.000 0.002 0.000 {method 'append' of 'list' objects}
1983 0.002 0.000 0.002 0.000 {method 'join' of 'str' objects}
2475 0.002 0.000 0.002 0.000 {method 'split' of 'str' objects}
3700 0.002 0.000 0.002 0.000 sql.py:2542(get_evolution)
3967 0.001 0.000 0.002 0.000 encoding.py:85(force_bytes)
495 0.002 0.000 0.002 0.000 locket.py:153(__init__)
495 0.001 0.000 0.002 0.000 weakref.py:345(__new__)
3966 0.002 0.000 0.002 0.000 {method 'tobytes' of 'memoryview' objects}
990 0.002 0.000 0.002 0.000 formdata.py:695(<listcomp>)
1486 0.002 0.000 0.002 0.000 workflows.py:690(get_backoffice_fields)
991 0.001 0.000 0.002 0.000 storage.py:73(fix_key)
495 0.002 0.000 0.002 0.000 weakref.py:350(__init__)
2476 0.002 0.000 0.002 0.000 formdef.py:641(get_workflow)
1 0.000 0.000 0.002 0.002 storage.py:574(get)
1 0.000 0.000 0.002 0.002 storage.py:682(get_filename)
495 0.001 0.000 0.002 0.000 posixpath.py:140(basename)
2475 0.002 0.000 0.002 0.000 {built-in method __new__ of type object at 0x906da0}
4952 0.001 0.000 0.001 0.000 {built-in method posix.fspath}
6868 0.001 0.000 0.001 0.000 {method 'getrandbits' of '_random.Random' objects}
495 0.001 0.000 0.001 0.000 tempfile.py:224(_infer_return_type)
1980 0.001 0.000 0.001 0.000 {method 'match' of 're.Pattern' objects}
495 0.001 0.000 0.001 0.000 {built-in method posix.getpid}
495 0.001 0.000 0.001 0.000 carddata.py:59(get_display_label)
495 0.001 0.000 0.001 0.000 locket.py:137(acquire)
990 0.001 0.000 0.001 0.000 sql.py:468(<listcomp>)
6930 0.001 0.000 0.001 0.000 parse.py:100(_noop)
496 0.001 0.000 0.001 0.000 formdef.py:579(get_all_fields)
6437 0.001 0.000 0.001 0.000 {built-in method builtins.len}
495 0.001 0.000 0.001 0.000 locket.py:148(release)
990 0.001 0.000 0.001 0.000 formdata.py:816(get_display_id)
990 0.001 0.000 0.001 0.000 encoding.py:51(force_text)
990 0.001 0.000 0.001 0.000 {method 'isoformat' of 'datetime.datetime' objects}
990 0.001 0.000 0.001 0.000 {method 'fileno' of '_io.TextIOWrapper' objects}
990 0.001 0.000 0.001 0.000 {built-in method _thread.allocate_lock}
2018 0.001 0.000 0.001 0.000 {method 'replace' of 'str' objects}
2972 0.001 0.000 0.001 0.000 {method 'endswith' of 'str' objects}
2 0.001 0.000 0.001 0.000 {method 'commit' of 'psycopg2.extensions.connection' objects}
1486 0.001 0.000 0.001 0.000 storage.py:446(get_table_name)
3966 0.001 0.000 0.001 0.000 <ipython-input-64-5bad67047637>:1(<lambda>)
495 0.000 0.000 0.001 0.000 storage.py:443(is_readonly)
495 0.001 0.000 0.001 0.000 codecs.py:186(__init__)
3960 0.001 0.000 0.001 0.000 {method 'bit_length' of 'int' objects}
495 0.001 0.000 0.001 0.000 {method 'write' of '_io.BufferedWriter' objects}
495 0.001 0.000 0.001 0.000 {built-in method _weakref._remove_dead_weakref}
990 0.001 0.000 0.001 0.000 {method 'rfind' of 'str' objects}
4948 0.001 0.000 0.001 0.000 {method 'items' of 'dict' objects}
1986 0.001 0.000 0.001 0.000 formdata.py:148(__init__)
495 0.001 0.000 0.001 0.000 locket.py:102(__init__)
1 0.000 0.000 0.001 0.001 extensions.py:118(getquoted)
495 0.001 0.000 0.001 0.000 workflows.py:1978(is_visible)
495 0.000 0.000 0.000 0.000 {method 'fileno' of '_io.BufferedWriter' objects}
1 0.000 0.000 0.000 0.000 sql.py:2594(<dictcomp>)
1980 0.000 0.000 0.000 0.000 {method 'end' of 're.Match' objects}
495 0.000 0.000 0.000 0.000 {method 'copy' of 'dict' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of '_io.BufferedReader' objects}
495 0.000 0.000 0.000 0.000 tempfile.py:364(_get_candidate_names)
495 0.000 0.000 0.000 0.000 formdata.py:473(get_parent)
495 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
495 0.000 0.000 0.000 0.000 {method 'acquire' of '_thread.lock' objects}
1 0.000 0.000 0.000 0.000 management.py:4287(<listcomp>)
495 0.000 0.000 0.000 0.000 sql.py:2536(__init__)
1 0.000 0.000 0.000 0.000 formdef.py:1950(storage_load)
1 0.000 0.000 0.000 0.000 {method '__exit__' of 'psycopg2.extensions.cursor' objects}
990 0.000 0.000 0.000 0.000 {built-in method builtins.issubclass}
1 0.000 0.000 0.000 0.000 extensions.py:121(<listcomp>)
495 0.000 0.000 0.000 0.000 {built-in method sys.audit}
495 0.000 0.000 0.000 0.000 {method '__exit__' of '_thread.lock' objects}
498 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects}
495 0.000 0.000 0.000 0.000 {method 'release' of '_thread.lock' objects}
1 0.000 0.000 0.000 0.000 extensions.py:126(<listcomp>)
1 0.000 0.000 0.000 0.000 {method 'close' of 'psycopg2.extensions.cursor' objects}
495 0.000 0.000 0.000 0.000 formdata.py:1236(set_last_update_time)
495 0.000 0.000 0.000 0.000 {built-in method psycopg2._psycopg.adapt}
1 0.000 0.000 0.000 0.000 {built-in method _pickle.load}
1 0.000 0.000 0.000 0.000 sql.py:2890(get_data_fields)
497 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
495 0.000 0.000 0.000 0.000 formdata.py:1167(workflow_merged_roles_dict)
1 0.000 0.000 0.000 0.000 formdef.py:218(migrate)
1 0.000 0.000 0.000 0.000 management.py:4266(__init__)
495 0.000 0.000 0.000 0.000 {method 'getquoted' of 'psycopg2.extensions.Int' objects}
1 0.000 0.000 0.000 0.000 storage.py:674(storage_load)
1 0.000 0.000 0.000 0.000 management.py:4142(__init__)
67 0.000 0.000 0.000 0.000 utf_8.py:15(decode)
2 0.000 0.000 0.000 0.000 sql.py:432(parse_clause)
1 0.000 0.000 0.000 0.000 afterjobs.py:57(__init__)
2 0.000 0.000 0.000 0.000 sql.py:506(get_connection_and_cursor)
14 0.000 0.000 0.000 0.000 fields.py:598(migrate)
34 0.000 0.000 0.000 0.000 sql.py:428(get_field_id)
1 0.000 0.000 0.000 0.000 uuid.py:713(uuid4)
67 0.000 0.000 0.000 0.000 {built-in method _codecs.utf_8_decode}
2 0.000 0.000 0.000 0.000 carddef.py:54(data_class)
8 0.000 0.000 0.000 0.000 fields.py:1294(migrate)
2 0.000 0.000 0.000 0.000 sql.py:476(get_connection)
1 0.000 0.000 0.000 0.000 {built-in method builtins.any}
1 0.000 0.000 0.000 0.000 uuid.py:138(__init__)
2 0.000 0.000 0.000 0.000 {method 'cursor' of 'psycopg2.extensions.connection' objects}
2 0.000 0.000 0.000 0.000 sql.py:204(as_sql)
4 0.000 0.000 0.000 0.000 fields.py:2126(migrate)
1 0.000 0.000 0.000 0.000 fields.py:2937(migrate)
29 0.000 0.000 0.000 0.000 storage.py:713(<genexpr>)
1 0.000 0.000 0.000 0.000 sql.py:2238(<listcomp>)
4 0.000 0.000 0.000 0.000 formdef.py:342(data_class_name)
2 0.000 0.000 0.000 0.000 sql.py:181(as_sql_param)
1 0.000 0.000 0.000 0.000 sql.py:2972(get_order_by_clause)
3 0.000 0.000 0.000 0.000 sql.py:114(__init__)
2 0.000 0.000 0.000 0.000 sql.py:119(as_sql)
1 0.000 0.000 0.000 0.000 {method 'join' of 'bytes' objects}
1 0.000 0.000 0.000 0.000 uuid.py:279(__str__)
4 0.000 0.000 0.000 0.000 {method 'title' of 'str' objects}
34 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 formdef.py:1942(__setstate__)
1 0.000 0.000 0.000 0.000 {built-in method posix.urandom}
1 0.000 0.000 0.000 0.000 sql.py:2182(get_objects)
1 0.000 0.000 0.000 0.000 encoder.py:104(__init__)
1 0.000 0.000 0.000 0.000 extensions.py:111(__init__)
4 0.000 0.000 0.000 0.000 {built-in method builtins.id}
1 0.000 0.000 0.000 0.000 {built-in method from_bytes}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 storage.py:440(__init__)
1 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'count' of 'list' objects}
2 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects}
4 0.000 0.000 0.000 0.000 {built-in method builtins.callable}
2 0.000 0.000 0.000 0.000 {built-in method builtins.globals}
1 0.000 0.000 0.000 0.000 {built-in method time.time}
1 0.000 0.000 0.000 0.000 sql.py:2189(get_order_by_clause)
1 0.000 0.000 0.000 0.000 extensions.py:115(prepare)
1 0.000 0.000 0.000 0.000 __init__.py:62(N_)
1 0.000 0.000 0.000 0.000 management.py:4284(<dictcomp>)
1 0.000 0.000 0.000 0.000 sql.py:2892(<listcomp>)
1 0.000 0.000 0.000 0.000 sql.py:2107(get_ids)
1 0.000 0.000 0.000 0.000 {method 'values' of 'dict' objects}
Mis à jour par Benjamin Dauvergne il y a plus d'un an
Benjamin Dauvergne a écrit :
En fait on perd du temps à écrire des fichiers sur le disque dans AfterJob.increment_count()
[...]
J'ai écrit une autre fonction qui ne passe pas par JsonFileExportAfterJob et là le gros du temps est perdu dans deep_bytes2str et preprocess_struct_time (genre 400ms sur 800), c'est pour après.
PS: sans passer par l'AfterJob, en enlevant evolution et rôles :
In [97]: def f():
...: items = list(carddef.data_class().select([Equal('fbod24ee8d0-8da6-4562-a8e4-f0eca4e0bd26', False)]))
...: carddef.data_class().load_all_evolutions(items)
...: prefetched_users = {str(user.id): user for user in User.select()}
...: prefetched_roles = {str(role.id): role for role in Role.select()}
...: return len(json.dumps([x.get_json_export_dict(anonymise=False, digest_key='default', prefetched_users=users, prefetched_roles=roles, include_evolution=False, include_roles=False, include_files=False
...: , include_unnamed_fields=False) for x in items], cls=JSONEncoder))
...:
...:
In [98]: cProfile.run('f()', sort=SortKey.CUMULATIVE)
1313426 function calls (1127044 primitive calls) in 1.284 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.284 1.284 {built-in method builtins.exec}
1 0.000 0.000 1.284 1.284 <string>:1(<module>)
1 0.001 0.001 1.284 1.284 <ipython-input-97-b42d54d58c4e>:1(f)
3703/3702 0.002 0.000 1.006 0.000 sql.py:629(f)
1 0.007 0.007 0.840 0.840 sql.py:2585(load_all_evolutions)
3966 0.021 0.000 0.813 0.000 sql.py:105(pickle_loads)
1986 0.009 0.000 0.786 0.000 sql.py:2568(_row2evo)
4052 0.557 0.000 0.577 0.000 {method 'load' of '_pickle.Unpickler' objects}
67813/3966 0.131 0.000 0.211 0.000 storage.py:109(deep_bytes2str)
10092/1986 0.012 0.000 0.179 0.000 storage.py:123(<listcomp>)
1 0.000 0.000 0.163 0.163 sql.py:2294(select)
496 0.000 0.000 0.162 0.000 sql.py:2227(select_iterator)
1 0.000 0.000 0.150 0.150 __init__.py:183(dumps)
1 0.002 0.002 0.150 0.150 misc.py:603(encode)
496 0.000 0.000 0.134 0.000 sql.py:2243(retrieve)
496 0.001 0.000 0.133 0.000 sql.py:2174(get_objects_iterator)
1 0.001 0.001 0.119 0.119 <ipython-input-97-b42d54d58c4e>:6(<listcomp>)
495 0.010 0.000 0.118 0.000 formdata.py:1342(get_json_export_dict)
112944/1 0.083 0.000 0.117 0.117 misc.py:586(preprocess_struct_time)
1 0.000 0.000 0.117 0.117 misc.py:593(<listcomp>)
495 0.009 0.000 0.102 0.000 sql.py:2863(_row2ob)
495 0.022 0.000 0.081 0.000 sql.py:2392(_row2obdata)
594264 0.071 0.000 0.071 0.000 {built-in method builtins.isinstance}
2 0.063 0.032 0.064 0.032 {method 'execute' of 'psycopg2.extensions.cursor' objects}
2483 0.019 0.000 0.039 0.000 {method 'fetchone' of 'psycopg2.extensions.cursor' objects}
990 0.002 0.000 0.035 0.000 formdata.py:804(get_url)
990 0.002 0.000 0.033 0.000 carddef.py:121(get_url)
1 0.000 0.000 0.031 0.031 encoder.py:182(encode)
990 0.001 0.000 0.031 0.000 publisher.py:164(get_backoffice_url)
1 0.028 0.028 0.030 0.030 encoder.py:204(iterencode)
990 0.006 0.000 0.027 0.000 parse.py:516(urljoin)
495 0.002 0.000 0.027 0.000 formdata.py:1262(get_display_name)
990 0.001 0.000 0.027 0.000 formdata.py:1332(get_json_dict)
990 0.019 0.000 0.025 0.000 formdata.py:1287(get_json_data_dict)
6062 0.012 0.000 0.020 0.000 publisher.py:50(find_class)
990 0.001 0.000 0.020 0.000 _json.py:164(typecast_json)
990 0.002 0.000 0.019 0.000 __init__.py:299(loads)
37519 0.014 0.000 0.019 0.000 encoding.py:51(force_text)
495 0.000 0.000 0.018 0.000 functional.py:202(__wrapper__)
495 0.001 0.000 0.017 0.000 functional.py:109(__init__)
39176/37691 0.011 0.000 0.017 0.000 sql.py:466(str_encode)
990 0.003 0.000 0.017 0.000 decoder.py:332(decode)
89735 0.016 0.000 0.017 0.000 {built-in method builtins.hasattr}
495 0.000 0.000 0.017 0.000 formdata.py:807(get_backoffice_url)
495 0.008 0.000 0.016 0.000 functional.py:125(__prepare_class__)
1980 0.004 0.000 0.013 0.000 parse.py:369(urlparse)
990 0.012 0.000 0.012 0.000 decoder.py:343(raw_decode)
2 0.000 0.000 0.011 0.005 storage.py:511(select)
13902 0.005 0.000 0.010 0.000 sql.py:2571(<genexpr>)
45697 0.009 0.000 0.009 0.000 {method 'get' of 'dict' objects}
52498 0.008 0.000 0.008 0.000 {built-in method builtins.id}
88 0.000 0.000 0.007 0.000 storage.py:526(<genexpr>)
6930 0.006 0.000 0.007 0.000 parse.py:111(_coerce_args)
86 0.000 0.000 0.007 0.000 storage.py:574(get)
495 0.001 0.000 0.007 0.000 functional.py:190(__mod__)
28402 0.006 0.000 0.006 0.000 {method 'startswith' of 'str' objects}
86 0.000 0.000 0.006 0.000 storage.py:682(get_filename)
990 0.001 0.000 0.006 0.000 parse.py:486(urlunparse)
495 0.001 0.000 0.006 0.000 functional.py:155(__text_cast)
495 0.002 0.000 0.006 0.000 formdata.py:1226(get_last_update_time)
495 0.001 0.000 0.005 0.000 __init__.py:41(gettext)
2611 0.005 0.000 0.005 0.000 {method 'timetuple' of 'datetime.datetime' objects}
495 0.002 0.000 0.005 0.000 formdata.py:708(get_visible_status)
1980 0.003 0.000 0.005 0.000 parse.py:434(urlsplit)
37519 0.005 0.000 0.005 0.000 {built-in method builtins.issubclass}
990 0.003 0.000 0.005 0.000 formdata.py:685(get_status)
6062 0.004 0.000 0.004 0.000 {built-in method builtins.__import__}
1485 0.001 0.000 0.004 0.000 publisher.py:148(get_frontoffice_url)
11385 0.003 0.000 0.004 0.000 {built-in method builtins.setattr}
495 0.000 0.000 0.003 0.000 __init__.py:78(gettext)
12357 0.003 0.000 0.003 0.000 {built-in method builtins.getattr}
2 0.000 0.000 0.003 0.002 storage.py:454(keys)
495 0.001 0.000 0.003 0.000 formdata.py:810(get_api_url)
990 0.002 0.000 0.003 0.000 parse.py:497(urlunsplit)
495 0.001 0.000 0.003 0.000 trans_real.py:343(gettext)
2 0.000 0.000 0.003 0.001 genericpath.py:16(exists)
2 0.003 0.001 0.003 0.001 {built-in method posix.stat}
86 0.003 0.000 0.003 0.000 {built-in method io.open}
15030 0.003 0.000 0.003 0.000 {method 'items' of 'dict' objects}
4052 0.002 0.000 0.002 0.000 encoding.py:85(force_bytes)
495 0.001 0.000 0.002 0.000 carddef.py:133(get_api_url)
990 0.001 0.000 0.002 0.000 misc.py:606(default)
1485 0.001 0.000 0.002 0.000 publisher.py:1012(get_cfg)
3816 0.001 0.000 0.002 0.000 monkeypatch.py:46(get_publisher)
3966 0.002 0.000 0.002 0.000 {method 'tobytes' of 'memoryview' objects}
86 0.000 0.000 0.002 0.000 storage.py:674(storage_load)
1980 0.002 0.000 0.002 0.000 {method 'match' of 're.Pattern' objects}
10923 0.002 0.000 0.002 0.000 {method 'add' of 'set' objects}
1980 0.001 0.000 0.001 0.000 <string>:1(<lambda>)
5445 0.001 0.000 0.001 0.000 carddata.py:25(get_formdef)
495 0.001 0.000 0.001 0.000 gettext.py:496(gettext)
990 0.001 0.000 0.001 0.000 sql.py:468(<listcomp>)
496 0.001 0.000 0.001 0.000 formdef.py:579(get_all_fields)
993 0.001 0.000 0.001 0.000 {method 'join' of 'str' objects}
3700 0.001 0.000 0.001 0.000 sql.py:2542(get_evolution)
990 0.001 0.000 0.001 0.000 {method 'isoformat' of 'datetime.datetime' objects}
6930 0.001 0.000 0.001 0.000 parse.py:100(_noop)
495 0.001 0.000 0.001 0.000 formdata.py:1623(__getattr__)
495 0.001 0.000 0.001 0.000 trans_real.py:101(get)
1986 0.001 0.000 0.001 0.000 formdata.py:148(__init__)
2476 0.001 0.000 0.001 0.000 formdef.py:641(get_workflow)
1486 0.001 0.000 0.001 0.000 workflows.py:690(get_backoffice_fields)
2 0.001 0.000 0.001 0.000 {method 'commit' of 'psycopg2.extensions.connection' objects}
4002 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
990 0.001 0.000 0.001 0.000 formdata.py:695(<listcomp>)
495 0.000 0.000 0.001 0.000 carddata.py:59(get_display_label)
1 0.000 0.000 0.001 0.001 extensions.py:118(getquoted)
175 0.000 0.000 0.001 0.000 posixpath.py:71(join)
1980 0.001 0.000 0.001 0.000 {method 'split' of 'str' objects}
89 0.000 0.000 0.001 0.000 storage.py:450(get_objects_dir)
1980 0.001 0.000 0.001 0.000 {built-in method __new__ of type object at 0x906da0}
990 0.000 0.000 0.000 0.000 formdata.py:816(get_display_id)
86 0.000 0.000 0.000 0.000 {built-in method builtins.any}
1 0.000 0.000 0.000 0.000 sql.py:2594(<dictcomp>)
1981 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1980 0.000 0.000 0.000 0.000 {method 'end' of 're.Match' objects}
495 0.000 0.000 0.000 0.000 sql.py:2536(__init__)
86 0.000 0.000 0.000 0.000 {method 'close' of '_io.BufferedReader' objects}
688 0.000 0.000 0.000 0.000 storage.py:713(<genexpr>)
1199 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects}
1 0.000 0.000 0.000 0.000 extensions.py:121(<listcomp>)
1 0.000 0.000 0.000 0.000 {method '__exit__' of 'psycopg2.extensions.cursor' objects}
1 0.000 0.000 0.000 0.000 {built-in method posix.listdir}
1 0.000 0.000 0.000 0.000 extensions.py:126(<listcomp>)
495 0.000 0.000 0.000 0.000 {method 'mro' of 'type' objects}
1 0.000 0.000 0.000 0.000 {method 'close' of 'psycopg2.extensions.cursor' objects}
495 0.000 0.000 0.000 0.000 workflows.py:1978(is_visible)
495 0.000 0.000 0.000 0.000 formdata.py:1236(set_last_update_time)
495 0.000 0.000 0.000 0.000 {built-in method psycopg2._psycopg.adapt}
495 0.000 0.000 0.000 0.000 formdata.py:473(get_parent)
1 0.000 0.000 0.000 0.000 sql.py:2890(get_data_fields)
495 0.000 0.000 0.000 0.000 formdata.py:1167(workflow_merged_roles_dict)
172 0.000 0.000 0.000 0.000 storage.py:73(fix_key)
497 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
495 0.000 0.000 0.000 0.000 {method 'getquoted' of 'psycopg2.extensions.Int' objects}
2 0.000 0.000 0.000 0.000 sql.py:506(get_connection_and_cursor)
175 0.000 0.000 0.000 0.000 posixpath.py:41(_get_sep)
2 0.000 0.000 0.000 0.000 sql.py:432(parse_clause)
67 0.000 0.000 0.000 0.000 utf_8.py:15(decode)
1 0.000 0.000 0.000 0.000 storage.py:460(<listcomp>)
2 0.000 0.000 0.000 0.000 {method 'cursor' of 'psycopg2.extensions.connection' objects}
2 0.000 0.000 0.000 0.000 carddef.py:54(data_class)
34 0.000 0.000 0.000 0.000 sql.py:428(get_field_id)
1 0.000 0.000 0.000 0.000 <ipython-input-97-b42d54d58c4e>:5(<dictcomp>)
175 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects}
86 0.000 0.000 0.000 0.000 roles.py:53(migrate)
86 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
175 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
67 0.000 0.000 0.000 0.000 {built-in method _codecs.utf_8_decode}
89 0.000 0.000 0.000 0.000 storage.py:446(get_table_name)
2 0.000 0.000 0.000 0.000 sql.py:181(as_sql_param)
4 0.000 0.000 0.000 0.000 formdef.py:342(data_class_name)
2 0.000 0.000 0.000 0.000 sql.py:476(get_connection)
2 0.000 0.000 0.000 0.000 sql.py:204(as_sql)
1 0.000 0.000 0.000 0.000 {method 'join' of 'bytes' objects}
3 0.000 0.000 0.000 0.000 sql.py:114(__init__)
2 0.000 0.000 0.000 0.000 sql.py:119(as_sql)
1 0.000 0.000 0.000 0.000 sql.py:2238(<listcomp>)
4 0.000 0.000 0.000 0.000 {method 'title' of 'str' objects}
34 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 extensions.py:111(__init__)
1 0.000 0.000 0.000 0.000 sql.py:2972(get_order_by_clause)
1 0.000 0.000 0.000 0.000 sql.py:2182(get_objects)
1 0.000 0.000 0.000 0.000 encoder.py:104(__init__)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 <ipython-input-97-b42d54d58c4e>:4(<dictcomp>)
4 0.000 0.000 0.000 0.000 {built-in method builtins.callable}
2 0.000 0.000 0.000 0.000 storage.py:481(sort_results)
2 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects}
1 0.000 0.000 0.000 0.000 extensions.py:115(prepare)
2 0.000 0.000 0.000 0.000 {built-in method builtins.globals}
1 0.000 0.000 0.000 0.000 sql.py:2892(<listcomp>)
1 0.000 0.000 0.000 0.000 sql.py:2189(get_order_by_clause)
Mis à jour par Lauréline Guérin il y a plus d'un an
mais pourquoi loader les evolutions si on n'en veut pas dans le résultat ?
Mis à jour par Benjamin Dauvergne il y a plus d'un an
Bizarre que ce soit la seule chose qui te saute aux yeux, mais bon il y a une réponse, parce que même si on en veut pas ça les charge quand même :
In [22]: def f():
...: items = list(carddef.data_class().select([Equal('fbod24ee8d0-8da6-4562-a8e4-f0eca4e0bd26', False)]))
...: users = {str(user.id): user for user in User.select()}
...: roles = {str(role.id): role for role in Role.select()}
...: return len(json.dumps([x.get_json_export_dict(anonymise=False, digest_key='default', prefetched_users=users, prefetched_roles=roles, include_evolution=False, include_roles=False, include_files=False
...: , include_unnamed_fields=False) for x in items], cls=JSONEncoder))
...:
In [23]: cProfile.run('f()', sort=SortKey.CUMULATIVE)
1324320 function calls (1137854 primitive calls) in 1.355 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.355 1.355 {built-in method builtins.exec}
1 0.000 0.000 1.355 1.355 <string>:1(<module>)
1 0.002 0.002 1.355 1.355 <ipython-input-22-6e5cfbf65bea>:1(f)
1 0.001 0.001 1.029 1.029 <ipython-input-22-6e5cfbf65bea>:5(<listcomp>)
495 0.017 0.000 1.028 0.002 formdata.py:1342(get_json_export_dict)
3702/3701 0.004 0.000 1.014 0.000 sql.py:629(f)
495 0.003 0.000 0.859 0.002 formdata.py:1226(get_last_update_time)
3700 0.012 0.000 0.850 0.000 sql.py:2542(get_evolution)
3966 0.031 0.000 0.382 0.000 sql.py:105(pickle_loads)
1986 0.018 0.000 0.372 0.000 sql.py:2568(_row2evo)
496 0.362 0.001 0.368 0.001 {method 'execute' of 'psycopg2.extensions.cursor' objects}
67885/3966 0.147 0.000 0.232 0.000 storage.py:109(deep_bytes2str)
10104/1986 0.012 0.000 0.199 0.000 storage.py:123(<listcomp>)
1 0.000 0.000 0.160 0.160 sql.py:2294(select)
496 0.000 0.000 0.160 0.000 sql.py:2227(select_iterator)
1 0.000 0.000 0.155 0.155 __init__.py:183(dumps)
1 0.002 0.002 0.155 0.155 misc.py:603(encode)
496 0.000 0.000 0.129 0.000 sql.py:2243(retrieve)
496 0.001 0.000 0.129 0.000 sql.py:2174(get_objects_iterator)
112944/1 0.084 0.000 0.119 0.119 misc.py:586(preprocess_struct_time)
1 0.000 0.000 0.119 0.119 misc.py:593(<listcomp>)
4052 0.087 0.000 0.115 0.000 {method 'load' of '_pickle.Unpickler' objects}
496 0.107 0.000 0.107 0.000 {method 'commit' of 'psycopg2.extensions.connection' objects}
495 0.009 0.000 0.101 0.000 sql.py:2863(_row2ob)
495 0.022 0.000 0.080 0.000 sql.py:2392(_row2obdata)
594504 0.074 0.000 0.074 0.000 {built-in method builtins.isinstance}
990 0.003 0.000 0.055 0.000 formdata.py:804(get_url)
990 0.003 0.000 0.051 0.000 carddef.py:121(get_url)
990 0.003 0.000 0.046 0.000 publisher.py:164(get_backoffice_url)
2977 0.021 0.000 0.040 0.000 {method 'fetchone' of 'psycopg2.extensions.cursor' objects}
990 0.009 0.000 0.039 0.000 parse.py:516(urljoin)
495 0.003 0.000 0.039 0.000 formdata.py:1262(get_display_name)
990 0.002 0.000 0.038 0.000 formdata.py:1332(get_json_dict)
990 0.028 0.000 0.036 0.000 formdata.py:1287(get_json_data_dict)
1 0.000 0.000 0.034 0.034 encoder.py:182(encode)
1 0.029 0.029 0.032 0.032 encoder.py:204(iterencode)
6062 0.017 0.000 0.028 0.000 publisher.py:50(find_class)
495 0.001 0.000 0.024 0.000 functional.py:202(__wrapper__)
495 0.001 0.000 0.023 0.000 functional.py:109(__init__)
89275 0.021 0.000 0.023 0.000 {built-in method builtins.hasattr}
495 0.010 0.000 0.022 0.000 functional.py:125(__prepare_class__)
495 0.001 0.000 0.021 0.000 formdata.py:807(get_backoffice_url)
37555 0.015 0.000 0.020 0.000 encoding.py:51(force_text)
1980 0.007 0.000 0.019 0.000 parse.py:369(urlparse)
39176/37691 0.012 0.000 0.019 0.000 sql.py:466(str_encode)
990 0.001 0.000 0.018 0.000 _json.py:164(typecast_json)
990 0.001 0.000 0.017 0.000 __init__.py:299(loads)
990 0.003 0.000 0.015 0.000 decoder.py:332(decode)
13902 0.006 0.000 0.013 0.000 sql.py:2571(<genexpr>)
495 0.002 0.000 0.012 0.000 functional.py:190(__mod__)
43711 0.012 0.000 0.012 0.000 {method 'get' of 'dict' objects}
990 0.011 0.000 0.011 0.000 decoder.py:343(raw_decode)
495 0.001 0.000 0.010 0.000 functional.py:155(__text_cast)
6930 0.008 0.000 0.010 0.000 parse.py:111(_coerce_args)
495 0.002 0.000 0.009 0.000 __init__.py:41(gettext)
2 0.000 0.000 0.009 0.004 storage.py:511(select)
2611 0.009 0.000 0.009 0.000 {method 'timetuple' of 'datetime.datetime' objects}
495 0.003 0.000 0.009 0.000 formdata.py:708(get_visible_status)
990 0.002 0.000 0.008 0.000 parse.py:486(urlunparse)
88 0.000 0.000 0.008 0.000 storage.py:526(<genexpr>)
52570 0.008 0.000 0.008 0.000 {built-in method builtins.id}
86 0.000 0.000 0.008 0.000 storage.py:574(get)
1980 0.004 0.000 0.007 0.000 parse.py:434(urlsplit)
28438 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
990 0.004 0.000 0.007 0.000 formdata.py:685(get_status)
86 0.001 0.000 0.007 0.000 storage.py:682(get_filename)
496 0.001 0.000 0.006 0.000 sql.py:506(get_connection_and_cursor)
4018 0.003 0.000 0.006 0.000 utf_8.py:15(decode)
495 0.001 0.000 0.006 0.000 __init__.py:78(gettext)
13838 0.006 0.000 0.006 0.000 {built-in method builtins.getattr}
6062 0.006 0.000 0.006 0.000 {built-in method builtins.__import__}
1485 0.002 0.000 0.006 0.000 publisher.py:148(get_frontoffice_url)
37555 0.005 0.000 0.005 0.000 {built-in method builtins.issubclass}
495 0.002 0.000 0.005 0.000 trans_real.py:343(gettext)
495 0.001 0.000 0.005 0.000 formdata.py:810(get_api_url)
4804 0.002 0.000 0.004 0.000 monkeypatch.py:46(get_publisher)
990 0.002 0.000 0.004 0.000 parse.py:497(urlunsplit)
15042 0.004 0.000 0.004 0.000 {method 'items' of 'dict' objects}
1485 0.002 0.000 0.003 0.000 publisher.py:1012(get_cfg)
86 0.003 0.000 0.003 0.000 {built-in method io.open}
11385 0.003 0.000 0.003 0.000 {built-in method builtins.setattr}
495 0.001 0.000 0.003 0.000 carddef.py:133(get_api_url)
496 0.003 0.000 0.003 0.000 {method 'cursor' of 'psycopg2.extensions.connection' objects}
4052 0.002 0.000 0.003 0.000 encoding.py:85(force_bytes)
4018 0.003 0.000 0.003 0.000 {built-in method _codecs.utf_8_decode}
993 0.002 0.000 0.002 0.000 {method 'join' of 'str' objects}
990 0.001 0.000 0.002 0.000 misc.py:606(default)
495 0.001 0.000 0.002 0.000 gettext.py:496(gettext)
3966 0.002 0.000 0.002 0.000 {method 'tobytes' of 'memoryview' objects}
1980 0.001 0.000 0.002 0.000 <string>:1(<lambda>)
5445 0.002 0.000 0.002 0.000 carddata.py:25(get_formdef)
10923 0.002 0.000 0.002 0.000 {method 'add' of 'set' objects}
86 0.000 0.000 0.002 0.000 storage.py:674(storage_load)
496 0.001 0.000 0.002 0.000 sql.py:476(get_connection)
495 0.002 0.000 0.002 0.000 formdata.py:1623(__getattr__)
495 0.001 0.000 0.001 0.000 trans_real.py:101(get)
1980 0.001 0.000 0.001 0.000 {method 'match' of 're.Pattern' objects}
496 0.001 0.000 0.001 0.000 formdef.py:579(get_all_fields)
1986 0.001 0.000 0.001 0.000 formdata.py:148(__init__)
990 0.001 0.000 0.001 0.000 sql.py:468(<listcomp>)
990 0.001 0.000 0.001 0.000 formdata.py:695(<listcomp>)
6930 0.001 0.000 0.001 0.000 parse.py:100(_noop)
990 0.001 0.000 0.001 0.000 {method 'isoformat' of 'datetime.datetime' objects}
1486 0.001 0.000 0.001 0.000 workflows.py:690(get_backoffice_fields)
2476 0.001 0.000 0.001 0.000 formdef.py:641(get_workflow)
4002 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
1980 0.001 0.000 0.001 0.000 {built-in method __new__ of type object at 0x906da0}
1980 0.001 0.000 0.001 0.000 {method 'split' of 'str' objects}
990 0.001 0.000 0.001 0.000 formdata.py:816(get_display_id)
2 0.000 0.000 0.001 0.000 storage.py:454(keys)
495 0.000 0.000 0.001 0.000 carddata.py:59(get_display_label)
495 0.001 0.000 0.001 0.000 {method 'close' of 'psycopg2.extensions.cursor' objects}
175 0.000 0.000 0.001 0.000 posixpath.py:71(join)
89 0.000 0.000 0.001 0.000 storage.py:450(get_objects_dir)
1 0.001 0.001 0.001 0.001 {built-in method posix.listdir}
86 0.000 0.000 0.000 0.000 {built-in method builtins.any}
1981 0.000 0.000 0.000 0.000 {built-in method builtins.len}
1199 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects}
495 0.000 0.000 0.000 0.000 {method 'mro' of 'type' objects}
495 0.000 0.000 0.000 0.000 workflows.py:1978(is_visible)
1980 0.000 0.000 0.000 0.000 {method 'end' of 're.Match' objects}
86 0.000 0.000 0.000 0.000 {method 'close' of '_io.BufferedReader' objects}
495 0.000 0.000 0.000 0.000 formdata.py:473(get_parent)
688 0.000 0.000 0.000 0.000 storage.py:713(<genexpr>)
495 0.000 0.000 0.000 0.000 sql.py:2536(__init__)
1 0.000 0.000 0.000 0.000 {method '__exit__' of 'psycopg2.extensions.cursor' objects}
495 0.000 0.000 0.000 0.000 formdata.py:1236(set_last_update_time)
1 0.000 0.000 0.000 0.000 sql.py:2890(get_data_fields)
172 0.000 0.000 0.000 0.000 storage.py:73(fix_key)
495 0.000 0.000 0.000 0.000 formdata.py:1167(workflow_merged_roles_dict)
496 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects}
175 0.000 0.000 0.000 0.000 posixpath.py:41(_get_sep)
2 0.000 0.000 0.000 0.000 genericpath.py:16(exists)
2 0.000 0.000 0.000 0.000 {built-in method posix.stat}
1 0.000 0.000 0.000 0.000 storage.py:460(<listcomp>)
2 0.000 0.000 0.000 0.000 sql.py:432(parse_clause)
34 0.000 0.000 0.000 0.000 sql.py:428(get_field_id)
175 0.000 0.000 0.000 0.000 {method 'endswith' of 'str' objects}
1 0.000 0.000 0.000 0.000 <ipython-input-22-6e5cfbf65bea>:4(<dictcomp>)
86 0.000 0.000 0.000 0.000 roles.py:53(migrate)
175 0.000 0.000 0.000 0.000 {built-in method posix.fspath}
86 0.000 0.000 0.000 0.000 {method 'encode' of 'str' objects}
1 0.000 0.000 0.000 0.000 carddef.py:54(data_class)
89 0.000 0.000 0.000 0.000 storage.py:446(get_table_name)
3 0.000 0.000 0.000 0.000 sql.py:114(__init__)
2 0.000 0.000 0.000 0.000 sql.py:204(as_sql)
2 0.000 0.000 0.000 0.000 formdef.py:342(data_class_name)
2 0.000 0.000 0.000 0.000 sql.py:181(as_sql_param)
1 0.000 0.000 0.000 0.000 sql.py:2238(<listcomp>)
2 0.000 0.000 0.000 0.000 sql.py:119(as_sql)
34 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects}
1 0.000 0.000 0.000 0.000 sql.py:2972(get_order_by_clause)
2 0.000 0.000 0.000 0.000 {method 'title' of 'str' objects}
1 0.000 0.000 0.000 0.000 sql.py:2182(get_objects)
1 0.000 0.000 0.000 0.000 encoder.py:104(__init__)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
2 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects}
1 0.000 0.000 0.000 0.000 <ipython-input-22-6e5cfbf65bea>:3(<dictcomp>)
4 0.000 0.000 0.000 0.000 {built-in method builtins.callable}
2 0.000 0.000 0.000 0.000 storage.py:481(sort_results)
2 0.000 0.000 0.000 0.000 {built-in method builtins.globals}
1 0.000 0.000 0.000 0.000 sql.py:2892(<listcomp>)
1 0.000 0.000 0.000 0.000 sql.py:2189(get_order_by_clause)
3700 appels à get_evolution qui coûte pas beaucoup plus cher que load_all_evolution vu que tout le temps est passé dans pickle&co, mais c'est un artefact de cProfile, en vrai ça va plus vite avec load_all_evolution():
In [24]: %timeit f() # sans load_all_evolution 959 ms ± 21.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ... In [26]: %timeit f() # avec load_all_evolution 434 ms ± 13.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Sur l'instance Toulouse les temps SQL varient beaucoup mais par contre les temps passés localement dans pickle&co sont à peu près stable sous cProfile, il faudrait placer un breakpoint dans get_evolution() pour voir qui l'appelle, cProfile ne donne pas le callgraph (ou alors je n'ai pas trouvé la bonne option).
Mis à jour par Benjamin Dauvergne il y a plus d'un an
Y a une méthode print_callers() bien utile sur l'objet pstat et donc les coupables sont:
- cette ligne où il faudrait inverse self.evolution et include_evolution pour bénéficier du court-circuitage :
if self.evolution and include_evolution: - get_visible_status() : là ce sera plus difficile de faire sans
- get_last_update_time() : logiquement on devrait pouvoir lire la valeur depuis la colonne mais le design est de toujours lire la valeur à jour depuis les évolutions, la colonne en ne sert qu'à filtrer, ceci pour que la valeur écrite soit toujours la dernière valeur à jour issue des évolutions (et le setter défini ne semble servir à rien j'ai l'impression)
PS: vu que c'est les pickle_loads qui coûtent cher et qu'ici on ne lit pas evo.parts ça pourrait être utile d'avoir du lazy loading à cet endroit.
Mis à jour par Pierre Ducroquet il y a plus d'un an
Pour le temps perdu sur les écritures disque, ne peut-on pas au moins à court terme tricher et modifier le filename du JsonFileExportAfterJob pour que ce soit /dev/shm/machin.json dans le cas des exports à destination de l'API ? Comme ça on va écrire en RAM, ça sera beaucoup plus rapide (à défaut de supprimer tous ces appels inutiles)
Ou écrire un JsonMemExportAfterJob (ou un truc mieux nommé) qui évite ça... Parce que là c'est quand même triste, surtout pour les cas où on est mal en IOs (HDS...)
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Assigné à mis à Benjamin Dauvergne
Je vais surtout supprimer ce increment_count dans le cas du web service.
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Fichier 0001-api-does-not-write-to-disk-during-form-card-listing-.patch 0001-api-does-not-write-to-disk-during-form-card-listing-.patch ajouté
- Statut changé de Nouveau à Solution proposée
- Patch proposed changé de Non à Oui
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Lié à Development #69108: api: performance du endpoint list, permettre d'exclure les evolutions ajouté
Mis à jour par Benjamin Dauvergne il y a plus d'un an
Voilà pour le gros des perfs, j'ai modifié localement sur toulouse-test-x (voir ticket #69080 pour tester) et ça charge le tableau en 2/3 secondes maintenant.
Mis à jour par Lauréline Guérin il y a plus d'un an
- Statut changé de Solution proposée à Solution validée
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Bloqué par Development #69109: api: performance du endpoint list, charger les parts de manière paresseuse ajouté
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Bloqué par Development #69109: api: performance du endpoint list, charger les parts de manière paresseuse supprimé
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Lié à Development #69109: api: performance du endpoint list, charger les parts de manière paresseuse ajouté
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Statut changé de Solution validée à Résolu (à déployer)
commit 0296a9e8a368e7336c2f22f1803739d22d68d3d0
Author: Benjamin Dauvergne <bdauvergne@entrouvert.com>
Date: Thu Sep 15 09:41:56 2022 +0200
api: does not write to disk during form/card listing (#69090)
The call to increment_count() is only beneficial to the JSON file export
case.Mis à jour par Benjamin Dauvergne il y a plus d'un an
Ça devrait accélérer le chargement des bases bijoe aussi et moins charger le partage NFS pendant ce temps.
Mis à jour par Pierre Ducroquet il y a plus d'un an
Benjamin Dauvergne a écrit :
Ça devrait accélérer le chargement des bases bijoe aussi et moins charger le partage NFS pendant ce temps.
Tout à fait, j'avais déjà vu de la charge sur le NFS que je ne m'expliquais pas. Je verrai après le déploiement ce que ça donne en prod à coup de fatrace...
Mis à jour par Benjamin Dauvergne il y a plus d'un an
- Lié à Development #69126: api: performance, l'appel à preprocess_struct_time dans JSONEncoder.encode est coûteux ajouté
Mis à jour par Transition automatique il y a plus d'un an
- Statut changé de Résolu (à déployer) à Solution déployée
api: does not write to disk during form/card listing (#69090)
The call to increment_count() is only beneficial to the JSON file export
case.