Improving Python API performance

Recently a MOSEK user pointed out out that sometimes the MOSEK Python API can be quite slow when it comes to input large amount of data. In particular the problem is related to the way MOSEK handles 32 and 64 bit integers.


If you use Numpy with a 64-bit Python, integer arrays will usually end up being arrays of numpy.int64 since the native Python integer is an int64.


Most MOSEK functions require arrays of numpy.int32, e.g. for variable or constraint indexing. In this case it can often significantly speed up the function call to explicitly convert the types.

To this extent is worth using numpy.array class to convert arrays of indexes to the right size: the numpy implementation is highly efficient and saves a lot of time!

Tests

To show the performance gain in Python 2.7 we implement a simple tests involving

• n=1000000 variables,
• linear objective function terms and
• quadratic objective function terms.

using Python timeit module. Each tests is repeated num=100 times and we removed from the timing unecessart contributions such as module import and task/environment declaration.

Note we show source code without timeit instrumentation to keep it simple. 

The complete testing code is available at the end of the post.

Objective function linear coefficents.

Say we want to input the coefficients for n variables in the objective function. We can do it using Python arrays or  numpy arrays instead. The code to time the running time is

	import mosek
	import numpy
	n = 1000000
	subi = numpy.array(range(n))
	val = numpy.ones(n,numpy.float64)
	env = mosek.Env()
	#input directly
	with env.Task() as task:
	task.appendvars(n)
	task.putclist(subi, val)
	#input using numpy conversion
	with env.Task() as task:
	task.appendvars(n)
	task.putclist(numpy.array(subi,numpy.int32), val)

view raw tmp.py hosted with ❤ by GitHub

We obtain the following results

• direct input: 1.113329s
• using numpy : 0.017299s

Objective function quadratic coefficients.

Say we want to input the coefficients of the linear part of the objective function. The benchmark code is

	import mosek
	import numpy
	n = 1000000
	qsubi = numpy.array(range(n))
	val = numpy.ones(n,numpy.float64)
	env= mosek.Env()
	#direct input
	with env.Task() as task:
	task.appendvars(n)
	task.putqobj(qsubi, qsubi, val)
	#using numpy
	with env.Task() as task:
	task.appendvars(n)
	task.putqobj(numpy.array(qsubi,numpy.int32), numpy.array(qsubi,numpy.int32), val)

view raw tmpq.py hosted with ❤ by GitHub

It provides the following results

• direct input: 2.239344s
• using numpy : 0.028208s

Conclusion

The MOSEK Python API converts by default 64bit integers to 32 bit ones, and this operation could introduce significant overhead. Using numpy array to convert integer arrays to 32 bit is a simple simple workaround that allows to speed up the input process significantly. 

The improvement is quite remarkable. However, it must be noticed that not all users will experience such an improvement. Nevertheless it is a good idea to check your code and see whether you can gain some performance.

Remember that this issue is solved in the new MOSEK 8 release.

Complete code

	import timeit
	num = 100
	n = 1000000
	setup_str="""
	import numpy
	import mosek
	n = %d
	qsubi = numpy.array(range(n))
	val = numpy.ones(n,numpy.float64)
	env = mosek.Env()
	"""%(n)
	print("Number of variables: %d"%n)
	print("Number of test repetitions: %d"%num)
	appendvars = """
	with env.Task() as task:
	task.appendvars(n)
	"""
	append_var_time=timeit.timeit(appendvars, setup=setup_str, number = 100)
	print("Time spent to append variables: %f" % (append_var_time/num) )
	putclist = """
	with env.Task() as task:
	task.appendvars(n)
	task.putclist(qsubi, val)
	"""
	putclist_time = timeit.timeit(putclist, setup=setup_str, number = num)
	print("Time spent to input objective function linear terms: %f" % ((putclist_time-append_var_time)/num) )
	putclist_conv = """
	with env.Task() as task:
	task.appendvars(n)
	task.putclist(numpy.array(qsubi,numpy.int32), val)
	"""
	putclist_time_rev=timeit.timeit(putclist_conv, setup=setup_str, number = num)
	print("Time spent to input objective function linear terms (revised): %f" % ((putclist_time_rev-append_var_time)/num) )
	putqobj = """
	with env.Task() as task:
	task.appendvars(n)
	task.putqobj(qsubi, qsubi, val)
	"""
	putqobj_time = timeit.timeit(putqobj, setup=setup_str, number = num)
	print("Time spent to input objective function quadratic terms: %f" % ((putqobj_time-append_var_time)/num))
	putqobj_conv = """
	with env.Task() as task:
	task.appendvars(n)
	task.putqobj(numpy.array(qsubi,numpy.int32), numpy.array(qsubi,numpy.int32), val)
	"""
	putqobj_time_rev = timeit.timeit(putqobj_conv, setup=setup_str, number = num)
	print("Time spent input objective function quadratic terms (revised): %f" % ((putqobj_time_rev-append_var_time)/num))

view raw timeit-python-np-array.py hosted with ❤ by GitHub