Here is a report about some of the talks I've attended at EuroPython this July. The conference had nine tracks running in parallel, the recordings of which are available online.
The conference was opened with a keynote by Django Girls organizers about the power of community, the progress made in just one year of Django Girls' existence, and their future plans. The slides for the presentation were illustrated with hand-drawn paintings and overall the keynote was quite inspiring.
Roberto Polli gave a talk "Scaling MySQL with Python." He made an overview of MySQL architecture, showed examples of using mysql-utilities for various kinds of administrative tasks, explained how to configure replication and failover setup and talked about MySQL Fabric, which is a Python framework for managing a MySQL deployment for sharding and high-availability.
Red Hatter Haikel Guemar gave a talk "The Lightweight Cloud Servers War Begins." The title basically says it all: Haikel explained why the need for lightweight operating systems for hosting containers emerged, gave an overview of the existing solutions (CoreOS, Project Atomic, Snappy Ubuntu, Photon, and Rancher), explained the role of their common components (systemd, etcd, cloud-init, kubernetes) and gave a general overview of the ecosystem evolving around containers. The talk aimed to answer the question "how it all fits together."
Francisco Fernández Castaño gave a talk "Knowing your garbage collector." Francisco explained the mechanism of garbage collection, described the approach CPython uses (reference counting and cycle detection), listed the problems associated with reference counting and explained the algorithms used for cycle detection. He also gave a quick overview of PyPy's current default (because yes, it's possible to choose a different GC for PyPy at the interpreter translation time) garbage collector–Minimark.
Xavier Fernandez talked about "Pip internals." The talk explained the implementation details of the way pip resolves dependencies, how it chooses the package to install and how it performs the installation.
At her "Metrics-driven development" talk, Lynn Root discussed the work her team at Spotify had done to enable their product to provide metrics for answering questions about the way it is being used by customers. The talk was somewhat abstract and more along the lines of generic ideas than practical guidelines.
A logical complement to Lynn's talk was given by Hynek Schlawack in "Beyond grep: Practical Logging and Metrics." He presented a very packed overview of popular tools for error reporting, logging, monitoring, and gathering metrics from a web service.
Timo Stollenwerk, a core developer of a popular CMS Plone and the leader of Plone's continuous integration and testing team presented his talk "The Butler and the Snake–Continuous Integration for Python." He told about basic practices making CI effective and gave an overview of the tools used: Jenkins, Buildout, Robot framework for acceptance testing, static code analysis utilities, various Jenkins plugins for notifications/scaling/etc.
This year's EuroPython had the Python BDFL himself among the speakers. Guido van Rossum presented his keynote "Python Now and in the Future." He first talked about the ever-favorite topic of python2 versus python3, then explained the complexities of bug handling in a large open source project, and then answered questions from the audience. Questions were asked about the GIL, PyPy, and Guido's preferences in programming languages/frameworks/libraries.
Another talk Guido gave was about one of the newest additions to the language, proposed by Guido himself in PEP 484: "Type Hints." The idea of adding optional static typing to python was first proposed 15 years ago but was abandoned, emerged again in 2007 as a compromise in form of function annotations, then at last year's EuroPython, Bob Ippolito talked about the mypy project in his keynote, which resulted in a discussion on python-ideas mailing list started by Guido and eventually in proposing and accepting the PEP. Annotating code for type checking will remain completely optional, and will be useful for larger and older codebases as a way of documenting the code and helping to prevent bugs early. The talk explains quite convincingly the reasoning behind having this functionality added to the language.
One more talk on the same topic was presented by a JetBrains developer Andrey Vlasovskikh. JetBrains is a company that develops PyCharm–a feature-rich professional IDE for Python, and Andrey was one of the active contributors to PEP 484. He showed simple examples of annotating a library API and explained the basics of type inference in tools like PyCharm (as Guido shared in his talk, almost half of the time PyCharm is unable to infer the type of an expression, so type hints would help in those cases). He also suggested to share stub files (it's possible to move annotations to a separate file) for libraries in the typeshed GitHub repo so that other people would benefit from them.
I was fortunate to be selected to participate in a three-hour event hosted by Google called "SRE Classroom." It was in a workshop format where teams were working on designing a distributed Pub/Sub system which conforms to given requirements of consistency, performance and availability. It was strongly hinted that in order to build such a system an implementation of Paxos protocol should be used. Three days before the conference all workshop participants had been sent a (quite large) reading list (which included the explanation of Paxos) to prepare for the workshop. The class handouts had some useful guidelines about tackling similar problems.
Vlad Temian and Manu Danci presented the project they have been working on and are currently actually using in production in their talk "gitfs - Building a Filesystem in Python." gitfs allows to mount a repository branch locally, and have any subsequent changes made to the files to be automatically committed to the remote. It is written purely in Python and uses fusepy and pygit2. To have acceptable performance on big repositories it uses three layers of cache. Resolving conflicts is done by always accepting local changes. A dedicated worker reduces the number of commits by grouping changes. Overall, the talk was quite interesting.
Paul Hallett talked about "What it's Really Like Building RESTful APIs with Django." Paul demonstrated some examples of existing web APIs and gave recommendations on how to improve them. He suggested using a well-known django-rest-framework, emphasized the significance of carefully designing the API before writing any code, stressed the importance of documentation and gave plenty of useful small hints on how to implement a quality RESTful API.
Vasilij Litvinov from Intel gave a talk "Tuning Python Applications Can Dramatically Increase Performance." Vasilij started by explaining why profiling is the right approach for optimizing performance, but mainly the talk was a presentation of an experimental statistical profiler with a rich UI. The profiler introduces very little overhead (compared to other existing non-statistical profilers) on the application, so the performance of the application doesn't change because of being profiled. It's possible to apply for the technical preview of the profiler, unfortunately there's no final plan so far to make it available to the public or opensource it. However, Vasilij disclosed that Intel is working on a proof-of-concept "accelerated Python" and that work would definitely be open sourced.
Holger Krekel's keynote "Towards a more effective, decentralized web" was a somewhat philosophical collection of thoughts about innovation in general and the future of the web. Holger talked about replacing obsolete standards and protocols and as an example he introduced The InterPlanetary File System project and explained the computer-science ideas behind it.
CPython core developer Larry Hastings gave a nice talk about "Python's Infamous GIL." The talk was a clear and comprehensive introduction to the topic. Larry presented a historical overview of CPython development, described CPython's memory management and explained why the GIL was added to CPython. He also talked about consequences and ramifications of having the GIL now that using multiple processor cores is the norm, and what problems it causes. In the end Larry described attempts that had been made in the past to remove the GIL from CPython, and shared his opinion about what it would really take to remove it.
PyPy contributor Romain Guillebert talked about "PyPy and the future of the Python ecosystem". The main point of his talk was that for most of the existing python projects it's very hard to switch to a different python implementation, such as PyPy, IronPython, or Jython, because CPython keeps its users captive with C extensions. He stated that the official API makes assumptions on how the VM is written, and that doesn't allow to use optimizations for better performance or remove the GIL (as does PyPy-STM, there was a talk about it too–"The GIL is dead: PyPy-STM"–but last year's talk on it was more in-depth, so I recommend it). He argued that incrementally improving the C API would benefit both CPython and alternative python implementations. As a short-term hackish solution for complete C extension support in PyPy Romain wrote a library called PyMetabiosis, which works by embedding CPython into PyPy. Romain appealed to library authors to use CFFI instead of C extension API, and to library users to prefer CFFI-backed libraries.
Our own Petr Viktorin gave an educating talk "Import Deep Dive." The talk was about some of the under-the-hood machinery of imports in python3.4+, rules of thumb for structuring imports in packages, explanation of where python searches for modules and what types of modules exist in python, and some other useful import-related details.
The weekend of EuroPython is the time for sprinting! This year I was sprinting on Django, and filed a pull request (which unfortunately hasn't been reviewed yet) for a bug in Django's ORM.