Project

General

Profile

Bug #9747

Improve performance of users export to ODS

Added by Frédéric Péters over 3 years ago. Updated 8 months ago.

Status:
Rejeté
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
22 Jan 2016
Due date:
% Done:

0%

Patch proposed:
No
Planning:
No

Description

While #9402 made it possible for CSV export to work in a reasonable time (13 seconds) the export to ODS on the same site (vincennes) still takes too much time and leads to a timeout (set at 60 seconds).

History

#1 Updated by Benjamin Dauvergne over 3 years ago

Bad news https://github.com/kennethreitz/tablib/issues/19

I think the only way forward would be to rewrite the ODS export using equivalent w.c.s. code. generating XML files directly.

#2 Updated by Benjamin Dauvergne over 3 years ago

Strangely the implementation in tablib at the date of the issue does not use openpyxl for ods export at all...

#3 Updated by Benjamin Dauvergne over 3 years ago

Completely strange behaviour:

In [14]: %timeit -n1 -r1 tablib.Dataset(*[(i,i) for i in range(1000)]).ods
1 loops, best of 1: 3.55 s per loop

In [15]: %timeit -n1 -r1 tablib.Dataset(*[(i,) for i in range(1000)]).ods
1 loops, best of 1: 297 ms per loop

But the code is just ugly1, it even seems false ( ws.addElement(table.TableColumn()) done on each row of each column ?!?).

1 https://github.com/kennethreitz/tablib/blob/develop/tablib/formats/_ods.py#L54

#4 Updated by Benjamin Dauvergne over 3 years ago

Big improvement:

--- /usr/lib/python2.7/dist-packages/tablib/formats/_ods.py    2014-10-13 23:44:09.000000000 +0200
+++ /tmp/_ods.py    2016-01-22 18:47:32.346140742 +0100
@@ -68,33 +68,18 @@
             except TypeError:
                 ## col is already unicode
                 pass
-            ws.addElement(table.TableColumn())

             # bold headers
             if (row_number == 1) and dataset.headers:
                 odf_row.setAttribute('stylename', bold)
-                ws.addElement(odf_row)
                 cell = table.TableCell()
                 p = text.P()
                 p.addElement(text.Span(text=col, stylename=bold))
                 cell.addElement(p)
                 odf_row.addElement(cell)
-
             # wrap the rest
             else:
-                try:
-                    if '\n' in col:
-                        ws.addElement(odf_row)
-                        cell = table.TableCell()
-                        cell.addElement(text.P(text=col))
-                        odf_row.addElement(cell)
-                    else:
-                        ws.addElement(odf_row)
                         cell = table.TableCell()
                         cell.addElement(text.P(text=col))
                         odf_row.addElement(cell)
-                except TypeError:
                     ws.addElement(odf_row)
-                    cell = table.TableCell()
-                    cell.addElement(text.P(text=col))
-                    odf_row.addElement(cell)

#5 Updated by Benjamin Dauvergne over 3 years ago

It seems that tablib and/or ods export have never been really used...

#6 Updated by Benjamin Dauvergne over 3 years ago

It still takes more thant 1Go of memory to build the ODS and the gunicorn worker is killed :/

#7 Updated by Mikaël Ates over 3 years ago

  • Target version set to 2.2.0

#9 Updated by Frédéric Péters 8 months ago

  • Status changed from Nouveau to Rejeté

L'export ODS a finalement été retiré, #26450.

Also available in: Atom PDF