Improving Python API performance

Recently a MOSEK user pointed out out that sometimes the MOSEK Python API can be quite slow when it comes to input large amount of data. In particular the problem is related to the way MOSEK handles 32 and 64 bit integers.


If you use Numpy with a 64-bit Python, integer arrays will usually end up being arrays of numpy.int64 since the native Python integer is an int64.


Most MOSEK functions require arrays of numpy.int32, e.g. for variable or constraint indexing. In this case it can often significantly speed up the function call to explicitly convert the types.

To this extent is worth using numpy.array class to convert arrays of indexes to the right size: the numpy implementation is highly efficient and saves a lot of time!

Tests

To show the performance gain in Python 2.7 we implement a simple tests involving

• n=1000000 variables,
• linear objective function terms and
• quadratic objective function terms.

using Python timeit module. Each tests is repeated num=100 times and we removed from the timing unecessart contributions such as module import and task/environment declaration.

Note we show source code without timeit instrumentation to keep it simple. 

The complete testing code is available at the end of the post.

Objective function linear coefficents.

Say we want to input the coefficients for n variables in the objective function. We can do it using Python arrays or  numpy arrays instead. The code to time the running time is

We obtain the following results

• direct input: 1.113329s
• using numpy : 0.017299s

Objective function quadratic coefficients.

Say we want to input the coefficients of the linear part of the objective function. The benchmark code is

It provides the following results

• direct input: 2.239344s
• using numpy : 0.028208s

Conclusion

The MOSEK Python API converts by default 64bit integers to 32 bit ones, and this operation could introduce significant overhead. Using numpy array to convert integer arrays to 32 bit is a simple simple workaround that allows to speed up the input process significantly. 

The improvement is quite remarkable. However, it must be noticed that not all users will experience such an improvement. Nevertheless it is a good idea to check your code and see whether you can gain some performance.

Remember that this issue is solved in the new MOSEK 8 release.

Complete code