Jump to: navigation, search

Difference between revisions of "PyMySQL evaluation"

Line 264: Line 264:
  
 
As stated before, PyMySQL has a really matter-of-fact and straightforward
 
As stated before, PyMySQL has a really matter-of-fact and straightforward
implementation.  The code is very readable and its easy to discern even
+
implementation.  The code is very readable and it's easy to discern even
 
unusual features such as the INSERT..VALUES rewriting.  As far as the design,
 
unusual features such as the INSERT..VALUES rewriting.  As far as the design,
 
there are lots of areas where performance suffers a bit at the Python level;
 
there are lots of areas where performance suffers a bit at the Python level;
Line 296: Line 296:
 
converter function.  These areas of functionality would appear to most
 
converter function.  These areas of functionality would appear to most
 
Python programmers to be reasonable and straightforward, however under
 
Python programmers to be reasonable and straightforward, however under
Python profiling the repeated calls to functions shows up very prominently;
+
Python profiling the repeated calls to functions shows up very prominently.
see Appendix A for the test suite and full performance results, where we
+
The nesting of multiple process/converter calls should at least be inlined
run the same calls under PyMySQL and MySQL-Connector-Python; the PyMySQL
+
so that as few isinstance()/type() checks as possible are performed
version uses 26850503 calls and MySQL-Connector-Python uses 11674074.
+
on a particular object.
  
In PyMySQL's defense, despite more than twice as many function calls on average,
+
In Appendix A, a short performance suite is run against PyMySQL,  
for some reason it is still outperforming MySQL-Connector-Python, leading
+
Mysql-Connector-Python, and MySQLdb.  The heavy function call count
me to suspect perhaps MySQL-connector is relying upon a try/except somewhere
+
is definitely apparent in PyMySQL, which logs 26,850,503 function calls
or otherwise doing something less efficiently; also, while PyMySQL used 526000
+
compared to MySQL-connector's 11,674,074 for the same operations.  For some
isinstance() calls, MySQL-Connector-Python is using almost that much as well.
+
very odd reason, PyMySQL still outperforms MySQL-connector timewise; this
 
+
is very unusual for a program with twice as many function calls, however
Regardless, PyMySQL is still far slower than a native library like MySQL-Python
+
one explaination could be that MySQL-connector is doing something
and can greatly improve its callcount.  MySQLdb is added at the end and
+
in particular very slow, such as relying upon catching an exception within
runs the whole test in only 47K function calls and an incredible .9 seconds,
+
a tight loop (exception throws in Python are extremely slow).  Both pure-Python
compared to PyMySQL's 16 seconds and MySQL-connectors 18 seconds.
+
libraries use similar numbers of isinstance() calls, about 500K.  
  
 +
The dismal truth of how inefficient both of these libraries are is
 +
apparent when we look at MySQLdb - as it is written in C and not pure
 +
Python, it runs the same operations in 0.9 seconds,
 +
a full 1600% faster than PyMySQL's 16 seconds and even more than MySQL-connector's
 +
18 seconds, and logs only 47,000 Python function calls.
  
 +
In any case, if we're to use PyMySQL, while it will never be anywhere near
 +
as fast as MySQLdb, the call counts and time spent for core operations can
 +
be greatly improved with some simple refactorings, which we may consider
 +
pull-requesting to the project. 
  
 +
  
 
=== History Documentation ===
 
=== History Documentation ===

Revision as of 01:33, 28 January 2015

PyMySQL Evaluation

This page will capture issues related to Openstack moving to the PyMySQL driver for MySQL/MariaDB dbapi access.

Rationale

While the MySQL-Python driver is a very mature and stable driver, it does not provide compatibility with either Python 3 or with eventlet monkeypatching (MySQL-Python can be monkeypatched with eventlet, but this feature disabled by default and missed in documentation). So OpenStack's usage of MySQL-Python, combined with the fact that concurrency is provided by eventlet, means that we currently have fully serialized database access within a single process, that is, only one database command occurs at a time within an Openstack Python process.

Drivers Under Consideration

The two drivers that are known to provide eventlet-monkeypatch compatibility are MySQL-Connector and PyMySQL, as they are written in pure Python. The two other well-known drivers for MySQL are MySQL-Python and OurSQL, both of which are written in C and offer no explicit support for eventlet or async. They can reportedly be built to support gevent using a system called Greenify, but the maturity and/or stabiltiy of this system is unknown.

A comparison at http://www.diamondtin.com/2014/sqlalchemy-gevent-mysql-python-drivers-comparison/ illustrates performance metrics observed with all four of these drivers, with MySQL-python built both without and with the "greenify" system.

A summary of the status of all four drivers is as follows.

MySQL-Python

MySQL-Python is the most widely used Python driver. However, it does not support Python 3, and it does not support async systems unless used within a thread pool. While a system known as Greenify can potentially resolve the latter situation, the Python 3 limitation is still a deal-breaker - pull request with Python 3 support was proposed to MySQL-python in April 2014 and still not merged.

mysqlclient

mysqlclient is a fork of MySQL-python. It add Python 3.3 support and merges some pull requests.

OurSQL

OurSQL is an alternative MySQL driver that is also written in C. It features fast performance and uses a different execution model than MySQL-python, using prepared statements. OurSQL does have a Python 3 port hosted on launchpad but is not integrated with OurSQL itself nor is it published on Pypi. Because it is written in C, it does not offer compatibility with async systems including eventlet-style monkeypatching.

Besides the shaky Python 3 support and lack of eventlet support, OurSQL is also not maintained, seeing its last master commit on 2012-06-05.

MySQL-Connector-Python

MySQL-connector-Python is a pure Python MySQL driver, and is now released under the auspices of the MySQL project itself, as owned by Oracle. MySQL-connector-Python supports both Python 3 as well as eventlet monkeypatching, and is well maintained. It is endorsed by Oracle as the official Oracle-supported driver for MySQL, so to that extent, it is in most ways Openstack's first choice in driver. However, Oracle refuses to publish MySQL-connector-Python on Pypi, which is critical to the Openstack infrastructure. Repeated attempts to communicate with Oracle in order to resolve this issue have not made any progress. Therefore, for this one unfortunate reason, MySQL-connector-Python will not have a place in the Openstack ecosystem unless this issue is resolved.

PyMySQL

PyMySQL is a pure Python MySQL driver, first written as a rough port of the MySQL-Python driver. PyMySQL meets all of Openstack's criterion for a driver: it is fully open source, hosted on Github, released on Pypi, is actively maintained, is written in pure Python so is eventlet-monkeypatch compatible, and is fully Python 3 compatible.

As this document is named "PyMySQL Evaluation", it should be apparent that this is the driver Openstack is currently leaning towards. However, it does have some minor code quality issues which hopefully can be addressed in some way; the section below titled "PyMySQL Code Review" will summarize the current state of the code. We are in fact leaning towards this driver because its the only one that actually meets all criteria. PyMySQL appears to be fairly widely used, especially considering its low profile, so it's possible that this is the case for many other users as well.


MySQL DB Drivers Comparison

Project PyPi hosted Eventlet friendly Python 3 compatibility Maturity and/or stability Comment
MySQL-Python Yes Partial No Yes Can be monkeypatched by eventlet, but only to enable thread pooling
mysqlclient Yes Partial Yes Yes MySQL-python fork, so should be similar to it
OurSQL Yes No Yes, but not Pypi hosted  ??? Has not seen commits/releases in two years
MySQL-Connector-Python No Yes Yes Yes The official Oracle-supported driver for MySQL
PyMySQL Yes Yes Yes Yes, with a proviso.

PyMySQL Code Review

PyMySQL started roughly as a pure Python port of MySQL-Python. The sections below will detail various aspects of the driver and the status of the code. Overall, the general tone of PyMySQL is one of code that was written in the spirit of pragmatism and immediate need; it is straightforward, relatively free of serious antipatterns and performs its task in a matter-of-fact way. However, it lacks polish and completeness in many areas that would normally be in better condition for a more mature project. These areas are all highly fixable, and it is hoped that these items can be shared with the current PyMySQL developers, through a combination of this document itself as well as new issues and pull requests on the PyMySQL tracker, so that this fairly promising library can be pushed to the next level. This would require that the developers are amenable to these improvements.

Coding Style / Pep8

PyMySQL's coding style is pretty good. The majority of it passes all flake8 tests, save for a few dozen whitespace issues and some long lines here and there within the core modules, and moreso within the tests which appear to be a bit more crufty than the core modules. A full run of flake8 produces only 128 errors, and with a default run of autopep8 against E1,E2,E3, we can fix all of them in one pass except for 38 remaining long line warnings. The code layout is mostly idiomatic and reasonable.

The codebase is probably short on docstrings. Most of the Python DBAPI that's public is very sparse, so the majority of functions and methods we see within the core modules here are not typically invoked from the outside. The methods that are part of the DBAPI spec do seem to have docstrings for the most part, although these docstrings are mostly extremely terse (example, docstring for cursor.execute(): "Execute a query"). The purpose of methods and functions is pretty easy to discern just by their names and implementations, however those who seek inline documentation as a means to understand a codebase will be very disappointed.

Test Coverage

The test coverage for PyMySQL is definitely lacking. A visual inspection of the test modules that are present will be surprising to most seasoned DBAPI users, as while there are about a dozen distinct test modules, each one is surprisingly short, most containing fewer than ten tests; for a full test run, there's only 126 tests; contrast this to MySQL-Connector-Python which has 531 tests.

An additional critical issue with the tests is that virtually all tests are live round-trip tests against a real database; there are no non-live unit tests of any kind, leaving codepaths that are not easily exercised on a generic MySQL database out in the cold. This means that features designed for particular versions of MySQL, particular datatypes, particular error condtions, and particular kinds of message packets that aren't created for real are simply not tested.

This lack of tests definitely shows when run with coverage. The test suite is designed to be run with unittest, and features a simple "runtests.py" front-end script, with no sign of any use of coverage tools anywhere, including in its tox.ini script. Running the suite with py.test plus py-test-cov produces an 85% coverage level overall, which isn't *too* terrible, but concerningly, the biggest gaps are in the three most critical modules, connections.py, converters.py and cursors.py:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
pymysql/__init__                                                     51      3    94%
pymysql/_compat                                                      14      4    71%
pymysql/_socketio                                                    72     30    58%
pymysql/charset                                                     228      3    99%
pymysql/connections                                                 827    188    77%
pymysql/constants/CLIENT                                             18      0   100%
pymysql/constants/COMMAND                                            32      0   100%
pymysql/constants/ER                                                471      0   100%
pymysql/constants/FIELD_TYPE                                         29      0   100%
pymysql/constants/FLAG                                               15      0   100%
pymysql/constants/SERVER_STATUS                                      10      0   100%
pymysql/constants/__init__                                            0      0   100%
pymysql/converters                                                  148     51    66%
pymysql/cursors                                                     273     37    86%
pymysql/err                                                          40      1    98%
pymysql/tests/__init__                                               14      5    64%
pymysql/tests/base                                                   25      2    92%
pymysql/tests/test_DictCursor                                        73      2    97%
pymysql/tests/test_SSCursor                                          56      6    89%
pymysql/tests/test_basic                                            186      4    98%
pymysql/tests/test_connection                                        57      8    86%
pymysql/tests/test_example                                           17      2    88%
pymysql/tests/test_issues                                           300     48    84%
pymysql/tests/test_load_local                                        39      2    95%
pymysql/tests/test_nextset                                           43      4    91%
pymysql/tests/thirdparty/__init__                                     7      5    29%
pymysql/tests/thirdparty/test_MySQLdb/__init__                        6      2    67%
pymysql/tests/thirdparty/test_MySQLdb/capabilities                  196     23    88%
pymysql/tests/thirdparty/test_MySQLdb/dbapi20                       423    121    71%
pymysql/tests/thirdparty/test_MySQLdb/test_MySQLdb_capabilities      73      6    92%
pymysql/tests/thirdparty/test_MySQLdb/test_MySQLdb_dbapi20          100      2    98%
pymysql/tests/thirdparty/test_MySQLdb/test_MySQLdb_nonstandard       57      3    95%
pymysql/times                                                        12      0   100%
pymysql/util                                                         14      7    50%
-------------------------------------------------------------------------------------
TOTAL                                                              3926    569    86%

The lack of coverage in converters.py refers to functions related to handling incoming bound values as well as outgoing result values. Key data escaping features such as escaping of dictionaries, sets, some unicode objects and byte objects as well as many pathways into date and time-processing functions are not covered.

In connections.py, uncovered features include various SSL and very legacy (e.g. version 3.23) MySQL features, but also some protocol parsing features.

In cursors.py, there is coverage for an elaborate and mostly undocumented performance-related feature that rewrites an INSERT statement to have an extended VALUES clause (_do_execute_many), however it fails to cover the very likely case where the function will need to break the input set into multiple chunks. The code which does this obviously works and looks perfectly fine, but coverage marks it with a "!":

>         for arg in args:
>             v = values % escape(arg, conn)
>             if isinstance(v, text_type):
>                 v = v.encode(encoding)
>             if len(sql) + len(v) + 1 > max_stmt_length:
!                 rows += self.execute(sql)
!                 sql = bytearray(prefix)
>             else:
>                 sql += b','
>             sql += v
>         rows += self.execute(sql)

The most prominent missing coverage in cursors.py seems to be for its "scrollable cursor" support.

The lack of unit-style non-live tests as well as the overall terseness of the tests is a major concern for PyMySQL. An additional concern is that in many areas, many dozens of individual "feature tests" are combined together into a single method, making it extremely difficult to identify and debug regressions as well as to expand upon the test suite. In test_SSCursor.py we have essentially one giant test with about fourteen individual assertions. In test_basic.py, test_datatypes attempts to run round-trip tests against about fourteen datatypes at once, though to its credit it appears to have more individualized tests for many of these datatypes as well.

This style of testing is common for newer projects where the tests are written in as much of an expedient way as possible; the author of this document will freely admit that his own SQLAlchemy still has a lot of tests like this as well (though at well over 5000 tests, that proportion has become fairly small :) ). The proportion of tests which are like this in PyMySQL, relative to its very low number of tests to start with, can be easily improved.

There is also a suite called "test_issues", which intends to accumulate tests against specific issues that have been reported. This isn't a bad idea, though it would be nice to see an effort made into categorizing these tests into the actual features they are testing, rather than a meaningless list of issue numbers, which could then form as the basis for new suites centered around those areas of functionality. The issues reported should be used as the inspiration for improvement of the test suite, rather than just another line-item to be filed away.


Library Documentation

As the Python DBAPI is already a well-documented API, and the users of PyMySQL are typically coming from MySQL-Python which is documented to some degree and for which the use contract is widely known, it seems unlikely that users of PyMySQL are eager for comprehensive documentation specific to this library.

That being said, as far as can be discerned, PyMySQL has absolutely no library documentation of any kind. There's a README which essentially refers to pep-249, an example.py in the root which shows a very basic connection / round trip, and that is absolutely it. In order to know anything about the API, specific parameters, etc., one either has to go off of the MySQL-Python documentation and assume PyMySQL also features the same parameter, or read the source code.

This has probably been perfectly fine for PyMySQL and its userbase, however for PyMySQL to enter the big time, it should really build up a rudimentary Sphinx build and publish it up to RTD. This would be very easy to get started, and once present, new documentation sections can be added iteratively.


Architecture and Performance

As stated before, PyMySQL has a really matter-of-fact and straightforward implementation. The code is very readable and it's easy to discern even unusual features such as the INSERT..VALUES rewriting. As far as the design, there are lots of areas where performance suffers a bit at the Python level; these areas could be greatly improved very easily with a little bit of attention, however it's seems likely that there hasn't been too much work done with Python performance profiling here (that is, using cProfile).

As we are dealing with a driver that is communicating with a low level protocol over a socket, it's critical that sending and receiving messages is done as efficiently as possible. The mechanisms used in PyMySQL rely heavily on standard Python objects, without the use of __slots__; this means that for individual messages on a socket, we are creating a heavyweight object, calling its __init__() method and a creating a new __dict__ each time; the most common object, OKPacketWrapper(), already wraps a MysqlPacket() object, so this is fairly wasteful; it also implements a `__getattr__` scheme to proxy attribute access from itself to the internal MysqlPacket; this is a very inefficient mode of operation in Python as it means a lookup must first fail on the OKPacketWrapper before it invokes the `__getattr__()` method and does another attribute lookup.

There are also lots of areas where the same fairly expensive operation is done more times than needed. In particular, when calling upon cursor.execute(), for parameters passed as a dictionary under Python 2K, the isinstance() builtin function will be called on the incoming arguments four times at a minimum, with two of each call against the exact same criteria. Additional logic will then call isinstance() on every key of the dictionary as well as at least twice on every value within the dictionary, in order to determine if the value needs to be converted to bytes as well as for some top-level escaping logic checks; the escaping logic then goes further with more checks for type() to match the object to a final converter function. These areas of functionality would appear to most Python programmers to be reasonable and straightforward, however under Python profiling the repeated calls to functions shows up very prominently. The nesting of multiple process/converter calls should at least be inlined so that as few isinstance()/type() checks as possible are performed on a particular object.

In Appendix A, a short performance suite is run against PyMySQL, Mysql-Connector-Python, and MySQLdb. The heavy function call count is definitely apparent in PyMySQL, which logs 26,850,503 function calls compared to MySQL-connector's 11,674,074 for the same operations. For some very odd reason, PyMySQL still outperforms MySQL-connector timewise; this is very unusual for a program with twice as many function calls, however one explaination could be that MySQL-connector is doing something in particular very slow, such as relying upon catching an exception within a tight loop (exception throws in Python are extremely slow). Both pure-Python libraries use similar numbers of isinstance() calls, about 500K.

The dismal truth of how inefficient both of these libraries are is apparent when we look at MySQLdb - as it is written in C and not pure Python, it runs the same operations in 0.9 seconds, a full 1600% faster than PyMySQL's 16 seconds and even more than MySQL-connector's 18 seconds, and logs only 47,000 Python function calls.

In any case, if we're to use PyMySQL, while it will never be anywhere near as fast as MySQLdb, the call counts and time spent for core operations can be greatly improved with some simple refactorings, which we may consider pull-requesting to the project.


History Documentation

Community Involvement

Appendix A - Performance Tests

import cProfile
import StringIO
import pstats
import contextlib


@contextlib.contextmanager
def profiled(dbapi):
    pr = cProfile.Profile()
    pr.enable()
    yield
    pr.disable()
    s = StringIO.StringIO()
    ps = pstats.Stats(pr, stream=s).sort_stats('cumulative')
    print "DBAPI:  %s" % dbapi
    ps.print_stats()
    # uncomment this to see who's calling what
    # ps.print_callers()
    print "total calls: %s" % s.getvalue()


import MySQLdb
import pymysql
from mysql import connector as mysqlconnector


def go(dbapi):
    conn = dbapi.connect(
        user='scott', passwd='tiger', host='localhost', db='test')
    cursor = conn.cursor()
    cursor.execute("""
CREATE TABLE IF NOT EXISTS test_things (
    x INTEGER,
    y VARCHAR(255),
    z FLOAT
) engine=InnoDB
""")
    cursor.execute("DELETE from test_things")

    with profiled(dbapi):
        for row in xrange(1000):
            cursor.execute(
                "INSERT INTO test_things (x, y, z) "
                "VALUES (%(x)s, %(y)s, %(z)s)",
                {"x": row, "y": "row number %d" % row,
                 "z": row * 4.57292, }
            )

        for x in xrange(500):
            cursor.execute(
                "select * from test_things")
            for row in cursor.fetchall():
                row[0], row[1], row[2]

    cursor.close()
    conn.close()

go(pymysql)
go(mysqlconnector)
go(MySQLdb)

DBAPI:  <module 'pymysql' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/__init__.pyc'>
total calls:          26850503 function calls in 16.017 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1500    0.010    0.000   16.016    0.011 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:107(execute)
     1500    0.003    0.000   15.976    0.011 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:271(_query)
     1500    0.003    0.000   15.952    0.011 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:708(query)
     1500    0.003    0.000   15.911    0.011 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:854(_read_query_result)
     1500    0.004    0.000   15.905    0.011 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1060(read)
      500    0.003    0.000   15.658    0.031 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1106(_read_result_packet)
      500    0.921    0.002   15.527    0.031 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1139(_read_rowdata_packet)
   500000    3.404    0.000    9.210    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1152(_read_row_from_packet)
  2012000    1.179    0.000    5.374    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:314(read_length_coded_string)
   504500    1.533    0.000    5.246    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:808(_read_packet)
  2014500    1.265    0.000    3.056    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:296(read_length_encoded_integer)
  3539500    2.469    0.000    2.782    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:248(read)
  1009000    0.795    0.000    2.300    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:833(_read_bytes)
  1009000    1.178    0.000    1.400    0.000 {method 'read' of '_io.BufferedReader' objects}
   508500    0.381    0.000    0.699    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/util.py:3(byte2int)
  5058000    0.468    0.000    0.468    0.000 {len}
   500500    0.219    0.000    0.453    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1098(_check_packet_is_eof)
   504500    0.211    0.000    0.380    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:342(check_error)
  2504000    0.259    0.000    0.259    0.000 {method 'append' of 'list' objects}
   501500    0.182    0.000    0.231    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:329(is_eof_packet)
     3500    0.009    0.000    0.222    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/_socketio.py:45(readinto)
  1022000    0.205    0.000    0.205    0.000 {_struct.unpack}
     3500    0.203    0.000    0.203    0.000 {method 'recv_into' of '_socket.socket' objects}
   526000    0.198    0.000    0.198    0.000 {isinstance}
  1501000    0.197    0.000    0.197    0.000 {method 'get' of 'dict' objects}
   504500    0.173    0.000    0.173    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:241(__init__)
   504500    0.169    0.000    0.169    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:339(is_error_packet)
  2014500    0.162    0.000    0.162    0.000 {ord}
      500    0.006    0.000    0.127    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1176(_get_descriptions)
     2000    0.003    0.000    0.088    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:361(__init__)
     2000    0.020    0.000    0.085    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:365(__parse_field_descriptor)
   500000    0.055    0.000    0.055    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py:301(through)
     1500    0.006    0.000    0.037    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:877(_execute_command)
     1500    0.003    0.000    0.027    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:848(_write_bytes)
     1500    0.002    0.000    0.024    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py:223(meth)
     1500    0.021    0.000    0.021    0.000 {method 'sendall' of '_socket.socket' objects}
     1500    0.020    0.000    0.021    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:278(_do_get_result)
     1000    0.003    0.000    0.019    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:97(_escape_args)
     4000    0.002    0.000    0.015    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:101(<genexpr>)
     1000    0.001    0.000    0.014    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1089(_read_ok_packet)
     3000    0.003    0.000    0.013    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:674(escape)
     1000    0.005    0.000    0.012    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:416(__init__)
     8000    0.011    0.000    0.011    0.000 {method 'decode' of 'str' objects}
     3500    0.002    0.000    0.006    0.000 {method '_checkReadable' of '_io._IOBase' objects}
     4000    0.002    0.000    0.006    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:128(<genexpr>)
     2000    0.003    0.000    0.005    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py:19(escape_item)
     7500    0.003    0.000    0.005    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:117(ensure_bytes)
     2000    0.003    0.000    0.005    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:386(description)
     5500    0.004    0.000    0.004    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:271(advance)
      500    0.002    0.000    0.004    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:442(__init__)
     1000    0.001    0.000    0.004    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:684(escape_string)
     3500    0.004    0.000    0.004    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/_socketio.py:87(readable)
     3500    0.003    0.000    0.003    0.000 {method '_checkClosed' of '_io._IOBase' objects}
     1000    0.001    0.000    0.003    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py:59(escape_string)
     1500    0.003    0.000    0.003    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1043(__init__)
     1500    0.001    0.000    0.003    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:94(nextset)
     4000    0.002    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:397(get_column_length)
     1000    0.002    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py:56(escape_float)
     1000    0.002    0.000    0.002    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}
     3000    0.002    0.000    0.002    0.000 {_struct.pack}
     6000    0.002    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:64(_get_db)
     1500    0.001    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:82(_nextset)
     1500    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/util.py:9(int2byte)
      500    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:246(fetchall)
     1500    0.001    0.000    0.001    0.000 {min}
     2500    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:326(is_ok_packet)
     2000    0.001    0.000    0.001    0.000 {method 'items' of 'dict' objects}
     1500    0.001    0.000    0.001    0.000 {getattr}
     1000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py:52(escape_int)
     1000    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:262(read_all)
     1500    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py:1056(__del__)
      500    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py:69(_check_executed)
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py:21(__exit__)
        1    0.000    0.000    0.000    0.000 test.py:7(profiled)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


DBAPI:  <module 'mysql.connector' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/__init__.pyc'>
total calls:          11674074 function calls in 18.316 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      500    0.611    0.001   17.825    0.036 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:820(fetchall)
      500    0.002    0.000   10.829    0.022 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:655(get_rows)
      500    1.505    0.003   10.825    0.022 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/protocol.py:292(read_text_result)
   504500    3.208    0.000    7.065    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/network.py:219(recv_plain)
   500000    3.398    0.000    6.297    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:359(row_to_python)
  1009000    3.039    0.000    3.039    0.000 {method 'recv_into' of '_socket.socket' objects}
   500000    2.003    0.000    2.283    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/utils.py:220(read_lc_string_list)
   500000    0.625    0.000    2.010    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:531(_STRING_to_python)
   502000    0.355    0.000    1.254    0.000 {method 'decode' of 'bytearray' objects}
   502000    0.212    0.000    0.899    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py:15(decode)
   502000    0.688    0.000    0.688    0.000 {_codecs.utf_8_decode}
   509500    0.498    0.000    0.498    0.000 {_struct.unpack_from}
     1500    0.009    0.000    0.490    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:452(execute)
   500000    0.482    0.000    0.482    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:413(_INT_to_python)
     1500    0.004    0.000    0.436    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:705(cmd_query)
   500000    0.352    0.000    0.352    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:405(_FLOAT_to_python)
   504500    0.326    0.000    0.326    0.000 {method 'extend' of 'bytearray' objects}
     1500    0.006    0.000    0.318    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:481(_send_cmd)
  2501500    0.273    0.000    0.273    0.000 {method 'append' of 'list' objects}
   500500    0.240    0.000    0.240    0.000 {method 'startswith' of 'bytearray' objects}
   524002    0.143    0.000    0.143    0.000 {isinstance}
     1500    0.010    0.000    0.114    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:613(_handle_result)
  1004500    0.109    0.000    0.109    0.000 {len}
   500000    0.086    0.000    0.086    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:204(description)
     2000    0.013    0.000    0.038    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/protocol.py:226(parse_column)
     1000    0.008    0.000    0.034    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:346(_process_params_dict)
     1500    0.006    0.000    0.034    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/network.py:118(send_plain)
     1500    0.002    0.000    0.023    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py:223(meth)
     1500    0.020    0.000    0.020    0.000 {method 'sendall' of '_socket.socket' objects}
     5000    0.007    0.000    0.014    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/catch23.py:79(struct_unpack)
     1000    0.001    0.000    0.014    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:562(_handle_ok)
    12000    0.011    0.000    0.011    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/utils.py:167(read_lc_string)
     1000    0.004    0.000    0.011    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/protocol.py:199(parse_ok)
     3000    0.005    0.000    0.009    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:152(to_mysql)
     3000    0.003    0.000    0.007    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:130(quote)
     1500    0.002    0.000    0.006    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/protocol.py:118(make_command)
     1500    0.004    0.000    0.006    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:406(_handle_result)
     3000    0.003    0.000    0.006    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:102(escape)
     1500    0.004    0.000    0.006    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/network.py:53(_prepare_packets)
     1000    0.002    0.000    0.005    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/protocol.py:252(parse_eof)
      500    0.001    0.000    0.005    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:579(_handle_eof)
     2000    0.003    0.000    0.004    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:545(_handle_server_status)
     1500    0.002    0.000    0.004    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/utils.py:53(int1store)
     3000    0.003    0.000    0.003    0.000 {method 'encode' of 'str' objects}
     9001    0.003    0.000    0.003    0.000 {method 'replace' of 'str' objects}
     1500    0.003    0.000    0.003    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:295(_reset_result)
     6027    0.003    0.000    0.003    0.000 {method 'format' of 'str' objects}
     2500    0.003    0.000    0.003    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/utils.py:296(read_lc_int)
     4500    0.002    0.000    0.002    0.000 {_struct.pack}
     3000    0.002    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:1221(_set_unread_result)
      500    0.001    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/protocol.py:219(parse_column_count)
      500    0.001    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:757(_handle_eof)
     1000    0.002    0.000    0.002    0.000 {repr}
     4527    0.001    0.000    0.001    0.000 {getattr}
     4000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/constants.py:34(flag_is_set)
     1000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:383(_handle_noresultset)
     4000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:1233(_get_unread_result)
     2001    0.001    0.000    0.001    0.000 {method 'items' of 'dict' objects}
     3000    0.001    0.000    0.001    0.000 {method 'lower' of 'str' objects}
      500    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:307(_have_unread_result)
     1000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:173(_str_to_mysql)
      500    0.001    0.000    0.001    0.000 {range}
     1000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:169(_float_to_mysql)
     1000    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/conversion.py:161(_int_to_mysql)
     1500    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/connection.py:1313(_get_getwarnings)
     1500    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:200(reset)
      500    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mysql/connector/cursor.py:397(_handle_resultset)
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.py:71(search_function)
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/ascii.py:41(getregentry)
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py:21(__exit__)
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.py:49(normalize_encoding)
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.py:77(__new__)
        1    0.000    0.000    0.000    0.000 {__import__}
        1    0.000    0.000    0.000    0.000 test.py:7(profiled)
        3    0.000    0.000    0.000    0.000 {method 'get' of 'dict' objects}
        1    0.000    0.000    0.000    0.000 {method 'split' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {method 'translate' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {method 'join' of 'str' objects}
        1    0.000    0.000    0.000    0.000 {built-in method __new__ of type object at 0x1001534e8}
        1    0.000    0.000    0.000    0.000 {hasattr}

DBAPI:  <module 'MySQLdb' from '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/__init__.pyc'>
total calls:          47503 function calls in 0.916 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1500    0.008    0.000    0.915    0.001 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:164(execute)
     1500    0.002    0.000    0.888    0.001 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:353(_query)
     1500    0.004    0.000    0.498    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:315(_do_query)
     1500    0.016    0.000    0.388    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:358(_post_get_result)
     1500    0.001    0.000    0.372    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:324(_fetch_row)
      500    0.371    0.001    0.371    0.001 {built-in method fetch_row}
     1500    0.008    0.000    0.359    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:142(_do_get_result)
     1500    0.002    0.000    0.348    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:351(_get_result)
     1500    0.346    0.000    0.346    0.000 {method 'store_result' of '_mysql.connection' objects}
     1500    0.134    0.000    0.134    0.000 {method 'query' of '_mysql.connection' objects}
     4000    0.002    0.000    0.010    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:184(<genexpr>)
     3000    0.002    0.000    0.008    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/connections.py:267(literal)
     1500    0.007    0.000    0.007    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:107(_warning_check)
     3000    0.002    0.000    0.006    0.000 {method 'escape' of '_mysql.connection' objects}
     1000    0.002    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/converters.py:81(Float2Str)
     6000    0.002    0.000    0.002    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:159(_get_db)
     2500    0.002    0.000    0.002    0.000 {isinstance}
     1000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/connections.py:202(string_literal)
      500    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:380(fetchall)
      500    0.001    0.000    0.001    0.000 {built-in method describe}
     1000    0.001    0.000    0.001    0.000 {method 'string_literal' of '_mysql.connection' objects}
     1500    0.001    0.000    0.001    0.000 {method 'affected_rows' of '_mysql.connection' objects}
     1500    0.001    0.000    0.001    0.000 {method 'insert_id' of '_mysql.connection' objects}
     1000    0.001    0.000    0.001    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/converters.py:69(Thing2Str)
     1500    0.000    0.000    0.000    0.000 {method 'info' of '_mysql.connection' objects}
     1500    0.000    0.000    0.000    0.000 {method 'warning_count' of '_mysql.connection' objects}
     1000    0.000    0.000    0.000    0.000 {method 'iteritems' of 'dict' objects}
      500    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/MySQLdb/cursors.py:103(_check_executed)
      500    0.000    0.000    0.000    0.000 {len}
      500    0.000    0.000    0.000    0.000 {built-in method field_flags}
        1    0.000    0.000    0.000    0.000 /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py:21(__exit__)
        1    0.000    0.000    0.000    0.000 test.py:7(profiled)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}