Jekyll2020-11-21T14:38:44+00:00https://shwina.github.io/feed.xmlAshwin SrinathMade with <b style="color: #f45;"><3</b> by <b><a href="https://github.com/niklasbuschmann">@niklasbuschmann</a></b>__get__ it right: controlling attribute access in Python2020-11-16T00:00:00+00:002020-11-16T00:00:00+00:00https://shwina.github.io/get-it-right<p>In Python, you use the dot (<code class="language-plaintext highlighter-rouge">.</code>) operator to access attributes of an object.
Normally, that isn’t something you have to give much thought to.
However, when you want to customize what happens during attribute access,
things can get complicated.</p>
<p>In this post, we’ll see how to control what happens when you use the dot operator.
Before we can talk about customizing attribute access,
we need to discuss two related topics:
<strong>class and instance attributes</strong>, and <strong>descriptors</strong>.
If both of those are familiar to you,
feel free to skip ahead.</p>
<ul id="markdown-toc">
<li><a href="#class-and-instance-attributes" id="markdown-toc-class-and-instance-attributes">Class and instance attributes</a></li>
<li><a href="#descriptors" id="markdown-toc-descriptors">Descriptors</a></li>
<li><a href="#customizing-attribute-access" id="markdown-toc-customizing-attribute-access">Customizing attribute access</a></li>
<li><a href="#example-1-returning-none-when-an-attribute-isnt-found" id="markdown-toc-example-1-returning-none-when-an-attribute-isnt-found">Example 1: returning <code class="language-plaintext highlighter-rouge">None</code> when an attribute isn’t found</a> <ul>
<li><a href="#overriding-__getattribute__---the-wrong-way" id="markdown-toc-overriding-__getattribute__---the-wrong-way">Overriding <code class="language-plaintext highlighter-rouge">__getattribute__()</code> - the wrong way</a></li>
<li><a href="#overriding-__getattribute__---the-less-wrong-but-still-wrong-way" id="markdown-toc-overriding-__getattribute__---the-less-wrong-but-still-wrong-way">Overriding <code class="language-plaintext highlighter-rouge">__getattribute__()</code> - the less wrong, but still wrong, way</a></li>
<li><a href="#using-__getattr__---the-right-approach" id="markdown-toc-using-__getattr__---the-right-approach">Using <code class="language-plaintext highlighter-rouge">__getattr__</code> - the right approach</a></li>
</ul>
</li>
<li><a href="#example-2---an-attribute-that-updates-itself-when-accessed" id="markdown-toc-example-2---an-attribute-that-updates-itself-when-accessed">Example 2 - an attribute that updates itself when accessed</a> <ul>
<li><a href="#using-a-non-data-descriptor" id="markdown-toc-using-a-non-data-descriptor">Using a non-data descriptor</a></li>
<li><a href="#using-a-data-descriptor" id="markdown-toc-using-a-data-descriptor">Using a data descriptor</a></li>
<li><a href="#what-about-property" id="markdown-toc-what-about-property">What about <code class="language-plaintext highlighter-rouge">property</code>?</a></li>
</ul>
</li>
<li><a href="#further-reading" id="markdown-toc-further-reading">Further reading</a></li>
</ul>
<h2 id="class-and-instance-attributes">Class and instance attributes</h2>
<p>There are two kinds of attributes in Python:
<em>class</em> and <em>instance</em> attributes.
In the following class,
<code class="language-plaintext highlighter-rouge">volume</code> is a class attribute,
and <code class="language-plaintext highlighter-rouge">line</code> is an instance attribute.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Speaker</span><span class="p">:</span>
<span class="n">volume</span> <span class="o">=</span> <span class="s">"low"</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">line</span> <span class="o">=</span> <span class="n">line</span>
<span class="k">def</span> <span class="nf">speak</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">line</span> <span class="k">if</span> <span class="bp">self</span><span class="p">.</span><span class="n">volume</span> <span class="o">==</span> <span class="s">"low"</span> <span class="k">else</span> <span class="n">line</span><span class="p">.</span><span class="n">upper</span><span class="p">()</span>
</code></pre></div></div>
<p>Each instance of the class has its own <code class="language-plaintext highlighter-rouge">line</code> attribute.
It’s easy to see why calling <code class="language-plaintext highlighter-rouge">a.speak()</code> below returns <code class="language-plaintext highlighter-rouge">"hello"</code>
while <code class="language-plaintext highlighter-rouge">b.speak()</code> returns <code class="language-plaintext highlighter-rouge">"goodbye"</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">a</span> <span class="o">=</span> <span class="n">Speaker</span><span class="p">(</span><span class="s">"hello"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">b</span> <span class="o">=</span> <span class="n">Speaker</span><span class="p">(</span><span class="s">"goodbye"</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">a</span><span class="p">.</span><span class="n">speak</span><span class="p">()</span>
<span class="s">"hello"</span>
<span class="o">>>></span> <span class="n">b</span><span class="p">.</span><span class="n">speak</span><span class="p">()</span>
<span class="s">"goodbye"</span>
</code></pre></div></div>
<p>The class attribute <code class="language-plaintext highlighter-rouge">volume</code> is shared by all instances.
If you change its value, that change is visible to both <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">Speaker</span><span class="p">.</span><span class="n">volume</span> <span class="o">=</span> <span class="s">"high"</span>
<span class="o">>>></span> <span class="n">a</span><span class="p">.</span><span class="n">speak</span><span class="p">()</span>
<span class="s">'HELLO'</span>
<span class="o">>>></span> <span class="n">b</span><span class="p">.</span><span class="n">speak</span><span class="p">()</span>
<span class="s">'GOODBYE'</span>
</code></pre></div></div>
<p>Each instance has an <em>instance dictionary</em> where instance
attributes are stored:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">a</span><span class="p">.</span><span class="n">__dict__</span>
<span class="p">{</span><span class="s">'line'</span><span class="p">:</span> <span class="s">'hello'</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">b</span><span class="p">.</span><span class="n">__dict__</span>
<span class="p">{</span><span class="s">'line'</span><span class="p">:</span> <span class="s">'goodbye'</span><span class="p">}</span>
</code></pre></div></div>
<p>Class attributes are stored in a <em>class dictionary</em>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">Speaker</span><span class="p">.</span><span class="n">__dict__</span>
<span class="n">mappingproxy</span><span class="p">({</span><span class="s">'__module__'</span><span class="p">:</span> <span class="s">'__main__'</span><span class="p">,</span>
<span class="s">'volume'</span><span class="p">:</span> <span class="s">'low'</span><span class="p">,</span>
<span class="s">'__init__'</span><span class="p">:</span> <span class="o"><</span><span class="n">function</span> <span class="n">__main__</span><span class="p">.</span><span class="n">Speaker</span><span class="p">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">line</span><span class="p">)</span><span class="o">></span><span class="p">,</span>
<span class="s">'speak'</span><span class="p">:</span> <span class="o"><</span><span class="n">function</span> <span class="n">__main__</span><span class="p">.</span><span class="n">Speaker</span><span class="p">.</span><span class="n">speak</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">></span><span class="p">,</span>
<span class="s">'__dict__'</span><span class="p">:</span> <span class="o"><</span><span class="n">attribute</span> <span class="s">'__dict__'</span> <span class="n">of</span> <span class="s">'Speaker'</span> <span class="n">objects</span><span class="o">></span><span class="p">,</span>
<span class="s">'__weakref__'</span><span class="p">:</span> <span class="o"><</span><span class="n">attribute</span> <span class="s">'__weakref__'</span> <span class="n">of</span> <span class="s">'Speaker'</span> <span class="n">objects</span><span class="o">></span><span class="p">,</span>
<span class="s">'__doc__'</span><span class="p">:</span> <span class="bp">None</span><span class="p">})</span>
</code></pre></div></div>
<p>It’s important to remember this distinction between
instance and class attributes (and where they are stored).</p>
<h2 id="descriptors">Descriptors</h2>
<p>Descriptors are an important concept related to attribute access.
A descriptor is a class that defines one or more of the following methods:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">__get__()</code>,</li>
<li><code class="language-plaintext highlighter-rouge">__set__()</code>,</li>
<li>or <code class="language-plaintext highlighter-rouge">__delete__()</code></li>
</ol>
<p>Below is a simple descriptor class. Its <code class="language-plaintext highlighter-rouge">__get__()</code> method always returns 0.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ZeroAttribute</span><span class="p">:</span>
<span class="s">"""
Attribute that is always 0
"""</span>
<span class="k">def</span> <span class="nf">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">owner</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">0</span>
</code></pre></div></div>
<p>Descriptors are only useful as class variables:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">ZeroAttribute</span><span class="p">()</span>
</code></pre></div></div>
<p>Accessing <code class="language-plaintext highlighter-rouge">Foo.x</code> will run its <code class="language-plaintext highlighter-rouge">__get__()</code> method:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">Foo</span><span class="p">.</span><span class="n">x</span> <span class="c1"># calls ZeroAttribute.__get__()
</span><span class="mi">0</span>
</code></pre></div></div>
<p>Even though <code class="language-plaintext highlighter-rouge">x</code> was defined like a class attribute,
you can also access <code class="language-plaintext highlighter-rouge">x</code> as an <em>instance</em> attribute,
which <em>also</em> invokes the <code class="language-plaintext highlighter-rouge">ZeroAttribute.__get__()</code> method:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">a</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">a</span><span class="p">.</span><span class="n">x</span> <span class="c1"># calls ZeroAttribute.__get__()
</span><span class="mi">0</span>
</code></pre></div></div>
<h3 class="no_toc" id="arguments-to-the-__get__-method">Arguments to the <code class="language-plaintext highlighter-rouge">__get__()</code> method</h3>
<p>The <code class="language-plaintext highlighter-rouge">__get__()</code> method accepts two arguments, <code class="language-plaintext highlighter-rouge">obj</code> and <code class="language-plaintext highlighter-rouge">owner</code>.</p>
<ul>
<li>
<p>If the <code class="language-plaintext highlighter-rouge">__get__()</code> method is called by acessing a _class__ attribute,
<code class="language-plaintext highlighter-rouge">obj</code> is set to <code class="language-plaintext highlighter-rouge">None</code>, and <code class="language-plaintext highlighter-rouge">owner</code> is set to the class.</p>
</li>
<li>
<p>If the <code class="language-plaintext highlighter-rouge">__get__()</code> method is called by accessing an <em>instance</em> attribute,
<code class="language-plaintext highlighter-rouge">obj</code> is set to the instance, and <code class="language-plaintext highlighter-rouge">owner</code> is set to the type of the instance.</p>
</li>
</ul>
<p>This lets you specify different behaviour
for class attribute access and instance attribute access.
For example, if you explicitly don’t want to allow class attribute access,
you can do something like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ZeroAttribute</span><span class="p">:</span>
<span class="s">"""
Attribute that is always 0
"""</span>
<span class="k">def</span> <span class="nf">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">owner</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="k">if</span> <span class="n">obj</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">AttributeError</span><span class="p">()</span> <span class="c1"># don't allow accessing as a class attribute
</span> <span class="k">return</span> <span class="mi">0</span>
<span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">ZeroAttribute</span><span class="p">()</span>
</code></pre></div></div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">Foo</span><span class="p">.</span><span class="n">x</span> <span class="c1"># accessing as class attribute, will raise
</span><span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">7</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__get__</span>
<span class="nb">AttributeError</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="n">a</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">a</span><span class="p">.</span><span class="n">x</span> <span class="c1"># accessing as instance attribute, OK
</span><span class="mi">0</span>
</code></pre></div></div>
<h3 class="no_toc" id="data-descriptors-and-non-data-descriptors">Data descriptors and non-data descriptors</h3>
<p>A descriptor that only defines <code class="language-plaintext highlighter-rouge">__get__()</code> is called a <em>non-data descriptor</em>.
A descriptor that also defines <code class="language-plaintext highlighter-rouge">__set__()</code> or <code class="language-plaintext highlighter-rouge">__delete__()</code> is called a <em>data</em> descriptor.
As you’ll see in the next section, this is an important difference to remember.</p>
<h2 id="customizing-attribute-access">Customizing attribute access</h2>
<p>Now that you know about class and instance attributes,
and a little bit about descriptors,
you’re ready to understand how attribute access <em>really</em> works in Python,
and how to customize it.</p>
<p>When you write <code class="language-plaintext highlighter-rouge">x.y</code>, the <code class="language-plaintext highlighter-rouge">x.__getattribute__()</code> method is invoked.
The default implementation of <code class="language-plaintext highlighter-rouge">__getattribute__()</code> does the following:</p>
<ul>
<li>First, it checks if <code class="language-plaintext highlighter-rouge">y</code> is a <em>data</em> descriptor.
If so, it returns the result of its <code class="language-plaintext highlighter-rouge">__get__()</code> method.</li>
<li>Next, it tries to find <code class="language-plaintext highlighter-rouge">'y'</code> in the instance dictionary of <code class="language-plaintext highlighter-rouge">x</code> and return it.</li>
<li>Next, it checks if <code class="language-plaintext highlighter-rouge">y</code> is a <em>non_data</em> descriptor.
If so, it returns the result of its <code class="language-plaintext highlighter-rouge">__get__()</code> method.</li>
<li>Next, it tries to find <code class="language-plaintext highlighter-rouge">'y'</code> in the class dictionary of the type of <code class="language-plaintext highlighter-rouge">x</code> and return it.</li>
<li>Finally, if none of the above worked, it raises an <code class="language-plaintext highlighter-rouge">AttributeError</code>.</li>
</ul>
<p>If <code class="language-plaintext highlighter-rouge">__getattribute__()</code> raises an <code class="language-plaintext highlighter-rouge">AttributeError</code>,
<code class="language-plaintext highlighter-rouge">x.__getattr__()</code> is called <em>if it is defined</em>.</p>
<p>So you can control the way attribute access works in a few different ways:</p>
<ul>
<li>Override <code class="language-plaintext highlighter-rouge">__getattribute__</code></li>
<li>Write a <code class="language-plaintext highlighter-rouge">__getattr__</code></li>
<li>Make the attribute a descriptor object</li>
</ul>
<p>Let’s look at a some examples that will help understand when to use which.</p>
<h2 id="example-1-returning-none-when-an-attribute-isnt-found">Example 1: returning <code class="language-plaintext highlighter-rouge">None</code> when an attribute isn’t found</h2>
<p>By default, accessing an attribute that doesn’t exist gives you an <code class="language-plaintext highlighter-rouge">AttributeError</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="p">...</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="p">...</span> <span class="bp">self</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">x</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">42</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">y</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="nb">AttributeError</span><span class="p">:</span> <span class="s">'MyClass'</span> <span class="nb">object</span> <span class="n">has</span> <span class="n">no</span> <span class="n">attribute</span> <span class="s">'y'</span>
</code></pre></div></div>
<p>Let’s say we want to get <code class="language-plaintext highlighter-rouge">None</code> (or some other custom behaviour) when we access
non-existent attributes.</p>
<h3 id="overriding-__getattribute__---the-wrong-way">Overriding <code class="language-plaintext highlighter-rouge">__getattribute__()</code> - the wrong way</h3>
<p>The first approach we might consider is overriding <code class="language-plaintext highlighter-rouge">__getattribute__()</code>.
Try to spot the problem with the code below:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">x</span>
<span class="k">def</span> <span class="nf">__getattribute__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">__dict__</span><span class="p">[</span><span class="n">name</span><span class="p">]</span>
<span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
</code></pre></div></div>
<p>If we try to access a non-existent attribute of a <code class="language-plaintext highlighter-rouge">MyClass</code> instance,
we get a <code class="language-plaintext highlighter-rouge">RecursionError</code>!</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">y</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">1</span><span class="p">,</span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">6</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__getattribute__</span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">6</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__getattribute__</span>
<span class="n">File</span> <span class="s">"<stdin>"</span><span class="p">,</span> <span class="n">line</span> <span class="mi">6</span><span class="p">,</span> <span class="ow">in</span> <span class="n">__getattribute__</span>
<span class="p">[</span><span class="n">Previous</span> <span class="n">line</span> <span class="n">repeated</span> <span class="mi">996</span> <span class="n">more</span> <span class="n">times</span><span class="p">]</span>
<span class="nb">RecursionError</span><span class="p">:</span> <span class="n">maximum</span> <span class="n">recursion</span> <span class="n">depth</span> <span class="n">exceeded</span>
</code></pre></div></div>
<p>The problem is this line in <code class="language-plaintext highlighter-rouge">MyClass.__getattribute__()</code>,
which itself calls <code class="language-plaintext highlighter-rouge">MyClass.__getattribute__()</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">__dict__</span><span class="p">[</span><span class="n">name</span><span class="p">]</span> <span class="c1"># calls self.__getattr__("__dict__") - infinite recursion!
</span></code></pre></div></div>
<h3 id="overriding-__getattribute__---the-less-wrong-but-still-wrong-way">Overriding <code class="language-plaintext highlighter-rouge">__getattribute__()</code> - the less wrong, but still wrong, way</h3>
<p>To prevent <code class="language-plaintext highlighter-rouge">MyClass.__getattribute__()</code> from calling itself recursively,
we could use the base class’, i.e., <code class="language-plaintext highlighter-rouge">object</code>’s implementation of
<code class="language-plaintext highlighter-rouge">__getattribute__()</code> instead. If that fails, we return <code class="language-plaintext highlighter-rouge">None</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="n">x</span>
<span class="k">def</span> <span class="nf">__getattribute__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">object</span><span class="p">.</span><span class="n">__getattribute__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">AttributeError</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">None</span>
</code></pre></div></div>
<p>This gives us the behaviour we want:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">42</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">y</span> <span class="c1"># no error, returns None
</span><span class="o">>>></span>
</code></pre></div></div>
<h3 id="using-__getattr__---the-right-approach">Using <code class="language-plaintext highlighter-rouge">__getattr__</code> - the right approach</h3>
<p>The more elegant solution is to write a <code class="language-plaintext highlighter-rouge">__getattr__()</code> method.
Recall that the default implementation of
<code class="language-plaintext highlighter-rouge">__getattribute__()</code> will raise <code class="language-plaintext highlighter-rouge">AttributeError</code>
when it doesn’t find an attribute.
When that happens, <code class="language-plaintext highlighter-rouge">__getattr__()</code> is called:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">x</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span><span class="n">x</span>
<span class="k">def</span> <span class="nf">__getattr__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">None</span>
</code></pre></div></div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span> <span class="c1"># OK - __getattribute__ will find attribue 'x'
</span><span class="mi">42</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">y</span> <span class="c1"># OK - __getattr__ is invoked and returns None
</span></code></pre></div></div>
<h2 id="example-2---an-attribute-that-updates-itself-when-accessed">Example 2 - an attribute that updates itself when accessed</h2>
<p>As another example,
consider writing an attribute that is updated every time you access it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">2</span>
</code></pre></div></div>
<h3 id="using-a-non-data-descriptor">Using a non-data descriptor</h3>
<p>When you want to control the access behaviour of a specific attribute,
a descriptor is generally the right tool for the job.</p>
<p>The <code class="language-plaintext highlighter-rouge">IncrementingAttribute.__get__()</code> method below returns 0 the first time
it is called for an instance.
Subsequently, it returns 1, 2, 3, etc.
It does this by storing an internal attribute <code class="language-plaintext highlighter-rouge">_value</code> in the instance.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">IncrementingAttribute</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">owner</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="c1"># rdon't allowaccessing as a class attribute
</span> <span class="k">if</span> <span class="n">obj</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">AttributeError</span><span class="p">()</span>
<span class="c1"># if accessing for the first time, return 0,
</span> <span class="c1"># otherwise, increment by 1 and return the result
</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="s">"_value"</span><span class="p">):</span>
<span class="n">obj</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="n">obj</span><span class="p">.</span><span class="n">_value</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">obj</span><span class="p">.</span><span class="n">_value</span>
<span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">IncrementingAttribute</span><span class="p">()</span>
</code></pre></div></div>
<p>The attribute <code class="language-plaintext highlighter-rouge">x</code> is updated each time it is accessed:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">1</span>
</code></pre></div></div>
<p>What happens if we “reset” the value of <code class="language-plaintext highlighter-rouge">x</code> and then try to access it?</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">0</span> <span class="c1"># reset to 0
</span><span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
</code></pre></div></div>
<p>Oh no! <code class="language-plaintext highlighter-rouge">x</code> has somehow lost the ability to update itself.
To understand why,
it’s first important to understand what happens when the following line is executed:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">obj</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">0</span>
</code></pre></div></div>
<p>Because <code class="language-plaintext highlighter-rouge">x</code> does not define a <code class="language-plaintext highlighter-rouge">__set__()</code> method,
this line will fall back to the default behaviour of setting an attribute.
That is, it will add an entry <code class="language-plaintext highlighter-rouge">'x'</code> to the instance dictionary of <code class="language-plaintext highlighter-rouge">obj</code>.
You can see this by inspecting the <code class="language-plaintext highlighter-rouge">__dict__</code> attribute before and after
setting the attribute <code class="language-plaintext highlighter-rouge">x</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">__dict__</span>
<span class="p">{</span><span class="s">'_value'</span><span class="p">:</span> <span class="mi">0</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">__dict__</span>
<span class="p">{</span><span class="s">'_value'</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">1</span> <span class="c1"># adds entry 'x' to obj.__dict__
</span><span class="p">{</span><span class="s">'_value'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s">'x'</span><span class="p">:</span> <span class="mi">1</span><span class="p">}</span>
</code></pre></div></div>
<p>Recall that <code class="language-plaintext highlighter-rouge">x</code> is a <em>non-data descriptor</em>, that is,
it only defines a <code class="language-plaintext highlighter-rouge">__get__()</code> method.
Because <code class="language-plaintext highlighter-rouge">__getattribute__()</code> looks in the instance dictionary <em>before</em>
checking for non-data descriptors,
it finds <code class="language-plaintext highlighter-rouge">x</code> in the instance dictionary and returns that.</p>
<h3 id="using-a-data-descriptor">Using a data descriptor</h3>
<p>The solution to the above problem is to also define a <code class="language-plaintext highlighter-rouge">__set__()</code> method in our descriptor class:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">IncrementingAttribute</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__get__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">owner</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="c1"># don't allowaccessing as a class attribute
</span> <span class="k">if</span> <span class="n">obj</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="nb">AttributeError</span><span class="p">()</span>
<span class="c1"># if accessing for the first time, return 0,
</span> <span class="c1"># otherwise, increment by 1 and return the result
</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="s">"_value"</span><span class="p">):</span>
<span class="n">obj</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="n">obj</span><span class="p">.</span><span class="n">_value</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">obj</span><span class="p">.</span><span class="n">_value</span>
<span class="k">def</span> <span class="nf">__set__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="n">obj</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span> <span class="o">-</span> <span class="mi">1</span>
<span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">IncrementingAttribute</span><span class="p">()</span>
</code></pre></div></div>
<p>Now, we get the expected reset behaviour:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">obj</span> <span class="o">=</span> <span class="n">MyClass</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">1</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">0</span>
<span class="o">>>></span> <span class="n">obj</span><span class="p">.</span><span class="n">x</span>
<span class="mi">1</span>
</code></pre></div></div>
<h3 id="what-about-property">What about <code class="language-plaintext highlighter-rouge">property</code>?</h3>
<p>You could also use a <code class="language-plaintext highlighter-rouge">property</code> to achieve the same result:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyClass</span><span class="p">:</span>
<span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">x</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s">"_value"</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">_value</span>
<span class="o">@</span><span class="n">x</span><span class="p">.</span><span class="n">setter</span>
<span class="k">def</span> <span class="nf">x</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_value</span> <span class="o">=</span> <span class="n">value</span> <span class="o">-</span> <span class="mi">1</span>
</code></pre></div></div>
<p>In fact, <code class="language-plaintext highlighter-rouge">@property</code> is really just a way to define a (data) descriptor!
So if you find yourself writing properties that all look the same
(either in the same class or across different classes),
that’s a sign that you should write a descriptor instead.</p>
<h2 id="further-reading">Further reading</h2>
<p>If you’re looking for a detailed “Introduction to descriptors”,
or examples of how descriptors can be used,
see the
<a href="https://docs.python.org/3/howto/descriptor.html">Descriptor HowTo</a>
by Raymond Hettinger.
It’s one of my favourite parts of the official Python docs!</p>In Python, you use the dot (.) operator to access attributes of an object. Normally, that isn’t something you have to give much thought to. However, when you want to customize what happens during attribute access, things can get complicated.Customizing the compiler and linker used by setuptools2020-08-27T00:00:00+00:002020-08-27T00:00:00+00:00https://shwina.github.io/custom-compiler-linker-extensions<p>This post is about how to customize the compiler and linker used
when building a C/C++ extension for Python using setuptools.</p>
<p>I recently ran into the problem of configuring setuptools to use a custom
compiler and linker, and these are my notes on how I did that.</p>
<p>As an example, here is a simple <code class="language-plaintext highlighter-rouge">setup.py</code> for building the extension <code class="language-plaintext highlighter-rouge">spam</code> from
<code class="language-plaintext highlighter-rouge">spammodule.c</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">setup</span>
<span class="n">setup</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s">"spam"</span><span class="p">,</span>
<span class="n">ext_modules</span><span class="o">=</span><span class="p">[</span><span class="n">Extension</span><span class="p">(</span><span class="s">"spam"</span><span class="p">,</span> <span class="n">sources</span><span class="o">=</span><span class="p">[</span><span class="s">"spammodule.c"</span><span class="p">])],</span>
<span class="n">zip_safe</span><span class="o">=</span><span class="bp">False</span>
<span class="p">)</span>
</code></pre></div></div>
<p>Invoking <code class="language-plaintext highlighter-rouge">setup.py</code>, we can see the compiler and flags used for building the extension.
On my Mac:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python setup.py build_ext <span class="nt">--inplace</span>
running build_ext
building <span class="s1">'*'</span> extension
creating build
creating build/temp.macosx-10.9-x86_64-3.7
gcc <span class="nt">-Wno-unused-result</span> <span class="nt">-Wsign-compare</span> <span class="nt">-Wunreachable-code</span> <span class="nt">-DNDEBUG</span> <span class="nt">-g</span> <span class="nt">-fwrapv</span> <span class="nt">-O3</span> ...
</code></pre></div></div>
<p>Using a different compiler than <code class="language-plaintext highlighter-rouge">gcc</code> is easy – just set the value
of the <code class="language-plaintext highlighter-rouge">CC</code> environment variable:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ CC</span><span class="o">=</span>g++ python setup.py build_ext <span class="nt">--inplace</span>
running build_ext
building <span class="s1">'spam'</span> extension
creating build
creating build/temp.macosx-10.9-x86_64-3.7
g++ <span class="nt">-Wno-unused-result</span> <span class="nt">-Wsign-compare</span> <span class="nt">-Wunreachable-code</span> <span class="nt">-DNDEBUG</span> <span class="nt">-g</span> <span class="nt">-fwrapv</span> <span class="nt">-O3</span> ...
</code></pre></div></div>
<p>The compiler has changed from <code class="language-plaintext highlighter-rouge">gcc</code> to <code class="language-plaintext highlighter-rouge">g++</code>, but we notice that the
arguments passed to the compiler remain the same.
This can be a problem if you want to use a compiler that doesn’t support those argument.</p>
<hr />
<p><strong>Where do the compiler and linker flags come from?</strong></p>
<p>If you’re wondering where the compiler flags being used
(<code class="language-plaintext highlighter-rouge">-Wno-unsed-result -Wsign-compare ...</code>) come from,
they are set in the configuration used to compile CPython – see
<a href="https://github.com/python/cpython/blob/master/configure.ac">here</a>.</p>
<p>You can use <code class="language-plaintext highlighter-rouge">distutils.sysconfig</code> to inspect their values:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">distutils.sysconfig</span>
<span class="o">>>></span> <span class="n">distutils</span><span class="p">.</span><span class="n">sysconfig</span><span class="p">.</span><span class="n">get_config_var</span><span class="p">(</span><span class="s">"CFLAGS"</span><span class="p">)</span>
</code></pre></div></div>
<p>Also see <code class="language-plaintext highlighter-rouge">distutils.sysconfig.get_config_vars()</code> to get the values of <em>all</em>
configuration options.</p>
<hr />
<p>You can override these flags entirely and pass your own flags using
the <code class="language-plaintext highlighter-rouge">extra_compile_args</code> and <code class="language-plaintext highlighter-rouge">extra_link_args</code> options, as follows:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">setuptools.command.build_ext</span> <span class="kn">import</span> <span class="n">build_ext</span>
<span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">Extension</span><span class="p">,</span> <span class="n">setup</span>
<span class="k">class</span> <span class="nc">custom_build_ext</span><span class="p">(</span><span class="n">build_ext</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">build_extensions</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># Override the compiler executables. Importantly, this
</span> <span class="c1"># removes the "default" compiler flags that would
</span> <span class="c1"># otherwise get passed on to to the compiler, i.e.,
</span> <span class="c1"># distutils.sysconfig.get_var("CFLAGS").
</span> <span class="bp">self</span><span class="p">.</span><span class="n">compiler</span><span class="p">.</span><span class="n">set_executable</span><span class="p">(</span><span class="s">"compiler_so"</span><span class="p">,</span> <span class="s">"g++"</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">compiler</span><span class="p">.</span><span class="n">set_executable</span><span class="p">(</span><span class="s">"compiler_cxx"</span><span class="p">,</span> <span class="s">"g++"</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">compiler</span><span class="p">.</span><span class="n">set_executable</span><span class="p">(</span><span class="s">"linker_so"</span><span class="p">,</span> <span class="s">"g++"</span><span class="p">)</span>
<span class="n">build_ext</span><span class="p">.</span><span class="n">build_extensions</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="p">[</span>
<span class="n">setup</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s">"spam"</span><span class="p">,</span>
<span class="n">ext_modules</span><span class="o">=</span><span class="p">[</span>
<span class="n">Extension</span><span class="p">(</span>
<span class="s">"spam"</span><span class="p">,</span>
<span class="n">sources</span><span class="o">=</span><span class="p">[</span><span class="s">"spammodule.c"</span><span class="p">],</span>
<span class="n">extra_compile_args</span><span class="o">=</span><span class="p">[</span><span class="s">"-arch"</span><span class="p">,</span> <span class="s">"x86_64"</span><span class="p">],</span>
<span class="n">extra_link_args</span><span class="o">=</span><span class="p">[</span><span class="s">"-undefined"</span><span class="p">,</span> <span class="s">"dynamic_lookup"</span><span class="p">]</span>
<span class="p">)</span>
<span class="p">],</span>
<span class="n">zip_safe</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
<span class="n">cmdclass</span><span class="o">=</span><span class="p">{</span><span class="s">"build_ext"</span><span class="p">:</span> <span class="n">custom_build_ext</span><span class="p">}</span>
<span class="p">)</span>
</code></pre></div></div>
<p>Now we see that we have full control over the flags passed
to the compiler and linker commands:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>python setup.py build_ext <span class="nt">--inplace</span>
running build_ext
building <span class="s1">'spam'</span> extension
creating build
creating build/temp.macosx-10.9-x86_64-3.7
g++ <span class="nt">-I</span>/Users/ashwin/miniconda3/envs/sci37/include/python3.7m <span class="nt">-c</span> spammodule.c <span class="nt">-o</span> build/temp.macosx-10.9-x86_64-3.7/spammodule.o <span class="nt">-arch</span> x86_64
clang: warning: treating <span class="s1">'c'</span> input as <span class="s1">'c++'</span> when <span class="k">in </span>C++ mode, this behavior is deprecated <span class="o">[</span><span class="nt">-Wdeprecated</span><span class="o">]</span>
creating build/lib.macosx-10.9-x86_64-3.7
g++ <span class="nt">-arch</span> x86_64 build/temp.macosx-10.9-x86_64-3.7/spammodule.o <span class="nt">-o</span> build/lib.macosx-10.9-x86_64-3.7/spam.cpython-37m-darwin.so <span class="nt">-undefined</span> dynamic_lookup
copying build/lib.macosx-10.9-x86_64-3.7/spam.cpython-37m-darwin.so -> ~
</code></pre></div></div>This post is about how to customize the compiler and linker used when building a C/C++ extension for Python using setuptools.Using pytest with Cython2020-08-13T00:00:00+00:002020-08-13T00:00:00+00:00https://shwina.github.io/cython-testing<p>This post is about using <code class="language-plaintext highlighter-rouge">pytest</code> for testing <code class="language-plaintext highlighter-rouge">cdef</code> functions in Cython.</p>
<h2 id="the-problem">The problem</h2>
<p>Consider the <code class="language-plaintext highlighter-rouge">primes</code> function below,
which returns the first <code class="language-plaintext highlighter-rouge">n</code> prime numbers as a C++ vector:</p>
<div class="language-cython highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: foo.pxd
</span>
<span class="k">cdef</span> <span class="kt">vector</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="nf">primes</span><span class="p">(</span><span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">nb_primes</span><span class="p">)</span>
</code></pre></div></div>
<p>How can we write tests for <code class="language-plaintext highlighter-rouge">primes</code>
in a way that they are automatically discovered and run
by a test runner such as <code class="language-plaintext highlighter-rouge">pytest</code>?</p>
<hr />
<p><strong>Note</strong>: the <code class="language-plaintext highlighter-rouge">.pxd</code> file above contains only the “definition” of <code class="language-plaintext highlighter-rouge">primes</code>.
Here is the actual implementation:</p>
<div class="language-cython highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: foo.pyx
</span>
<span class="c1"># distutils: language=c++
# distutils: extra_compile_args=-std=c++11
</span>
<span class="kn">from</span> <span class="nn">libcpp.vector</span> <span class="kn">cimport</span> <span class="n">vector</span>
<span class="k">cdef</span> <span class="kt">vector</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="nf">primes</span><span class="p">(</span><span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">nb_primes</span><span class="p">):</span>
<span class="k">cdef</span> <span class="kt">int</span> <span class="n">n</span><span class="p">,</span> <span class="n">i</span>
<span class="k">cdef</span> <span class="kt">vector</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="n">p</span>
<span class="n">p</span><span class="p">.</span><span class="nf">reserve</span><span class="p">(</span><span class="n">nb_primes</span><span class="p">)</span> <span class="c1"># allocate memory for 'nb_primes' elements.
</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">2</span>
<span class="k">while</span> <span class="n">p</span><span class="p">.</span><span class="nf">size</span><span class="p">()</span> <span class="o"><</span> <span class="n">nb_primes</span><span class="p">:</span> <span class="c1"># size() for vectors is similar to len()
</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">p</span><span class="p">:</span>
<span class="k">if</span> <span class="n">n</span> <span class="o">%</span> <span class="n">i</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">p</span><span class="p">.</span><span class="nf">push_back</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="c1"># push_back is similar to append()
</span> <span class="n">n</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="c1"># Vectors are automatically converted to Python
</span> <span class="c1"># lists when converted to Python objects.
</span> <span class="k">return</span> <span class="n">p</span>
</code></pre></div></div>
<hr />
<h2 id="writing-the-tests">Writing the tests</h2>
<p><code class="language-plaintext highlighter-rouge">cdef</code> functions can only be called from Cython code.
Thus, tests for <code class="language-plaintext highlighter-rouge">cdef</code> functions must be written in Cython.
Let’s write a very simple test for <code class="language-plaintext highlighter-rouge">primes</code> in a separate file <code class="language-plaintext highlighter-rouge">foo_tests.pyx</code>:</p>
<div class="language-cython highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: foo_tests.pyx
</span>
<span class="c1"># distutils: language=c++
# distutils: extra_compile_args=-std=c++11
</span>
<span class="kn">from</span> <span class="nn">foo</span> <span class="kn">cimport</span> <span class="n">primes</span>
<span class="k">def</span> <span class="nf">test_primes</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">expect</span><span class="p">):</span>
<span class="k">assert</span> <span class="nf">primes</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="p">[]</span>
</code></pre></div></div>
<p>Compiling both using Cython:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cythonize <span class="nt">-i</span> foo.pyx foo_tests.pyx
</code></pre></div></div>
<p>Note that at this point, running <code class="language-plaintext highlighter-rouge">pytest</code> doesn’t do anything:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pytest
<span class="o">======================</span> no tests ran <span class="k">in </span>0.02s <span class="o">======================</span>
</code></pre></div></div>
<h2 id="making-tests-discoverable">Making tests discoverable</h2>
<p><code class="language-plaintext highlighter-rouge">pytest</code> doesn’t know to look for tests in Cython modules.
To make our tests discoverable,
we can make them attributes of a Python module
<code class="language-plaintext highlighter-rouge">test_foo.py</code> as follows:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># file: test_foo.py
</span>
<span class="kn">import</span> <span class="nn">importlib</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="c1"># list of Cython modules containing tests
</span><span class="n">cython_test_modules</span> <span class="o">=</span> <span class="p">[</span><span class="s">"foo_tests"</span><span class="p">]</span>
<span class="k">for</span> <span class="n">mod</span> <span class="ow">in</span> <span class="n">cython_test_modules</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># For each callable in `mod` with name `test_*`,
</span> <span class="c1"># set the result as an attribute of this module.
</span> <span class="n">mod</span> <span class="o">=</span> <span class="n">importlib</span><span class="p">.</span><span class="n">import_module</span><span class="p">(</span><span class="n">mod</span><span class="p">)</span>
<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="nb">dir</span><span class="p">(</span><span class="n">mod</span><span class="p">):</span>
<span class="n">item</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">mod</span><span class="p">,</span> <span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">callable</span><span class="p">(</span><span class="n">item</span><span class="p">)</span> <span class="ow">and</span> <span class="n">name</span><span class="p">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"test_"</span><span class="p">):</span>
<span class="nb">setattr</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">modules</span><span class="p">[</span><span class="n">__name__</span><span class="p">],</span> <span class="n">name</span><span class="p">,</span> <span class="n">item</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">ImportError</span><span class="p">:</span>
<span class="k">pass</span>
</code></pre></div></div>
<p>Now, our tests are discovered by <code class="language-plaintext highlighter-rouge">pytest</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pytest
<span class="o">=======================</span> <span class="nb">test </span>session starts <span class="o">=======================</span>
collected 0 items
<span class="o">=====================</span> warnings summary <span class="o">============================</span>
foo_tests.pyx:19
foo_tests.pyx:19: PytestCollectionWarning: cannot collect <span class="s1">'test_primes'</span> because it is not a <span class="k">function</span><span class="nb">.</span>
def test_primes<span class="o">(</span>data, expect<span class="o">)</span>:
<span class="nt">--</span> Docs: https://docs.pytest.org/en/latest/warnings.html
<span class="o">======================</span> 1 warning <span class="k">in </span>0.02s <span class="o">=========================</span>
</code></pre></div></div>
<p>…but, they aren’t actually run.</p>
<h2 id="making-tests-runnable">Making tests runnable</h2>
<p>A function from a Cython module is not a “regular” Python function,
and <code class="language-plaintext highlighter-rouge">pytest</code> refuses to collect it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">inspect</span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">foo_tests</span> <span class="kn">import</span> <span class="n">test_primes</span>
<span class="o">>>></span> <span class="nb">type</span><span class="p">(</span><span class="n">test_primes</span><span class="p">)</span>
<span class="n">_cython_3_0a5</span><span class="p">.</span><span class="n">cython_function_or_method</span>
<span class="o">>>></span> <span class="n">inspect</span><span class="p">.</span><span class="n">isfunction</span><span class="p">(</span><span class="n">test_primes</span><span class="p">)</span>
<span class="bp">False</span>
</code></pre></div></div>
<p>To get around this, we can wrap <code class="language-plaintext highlighter-rouge">test_primes</code> in a simple decorator that turns it
into a plain Python function:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">cytest</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="s">"""
Wraps `func` in a plain Python function.
"""</span>
<span class="o">@</span><span class="n">functools</span><span class="p">.</span><span class="n">wraps</span><span class="p">(</span><span class="n">func</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">wrapped</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">bound</span> <span class="o">=</span> <span class="n">inspect</span><span class="p">.</span><span class="n">signature</span><span class="p">(</span><span class="n">func</span><span class="p">).</span><span class="n">bind</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">bound</span><span class="p">.</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">bound</span><span class="p">.</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">wrapped</span>
</code></pre></div></div>
<p>Wrapping <code class="language-plaintext highlighter-rouge">test_primes</code> with the <code class="language-plaintext highlighter-rouge">cytest</code> decorator, we see that it now behaves
like a regular Python function, and thus is collected and run by <code class="language-plaintext highlighter-rouge">pytest</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">cytest</span>
<span class="k">def</span> <span class="nf">test_primes</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">expect</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">primes</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="p">[]</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>>>> from foo_tests import test_primes
>>> inspect.isfunction(test_primes)
True
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pytest
<span class="o">=======================</span> <span class="nb">test </span>session starts <span class="o">=======================</span>
test_foo.py <span class="nb">.</span>
<span class="o">=======================</span> 1 passed <span class="k">in </span>0.02s <span class="o">=========================</span>
</code></pre></div></div>
<h2 id="parametrizing-tests">Parametrizing tests</h2>
<p>In case you’re wondering: Yes! It is possible to parametrize tests with this
method:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">@</span><span class="n">pytest</span><span class="p">.</span><span class="n">mark</span><span class="p">.</span><span class="n">parametrize</span><span class="p">(</span>
<span class="s">"data,expect"</span><span class="p">,</span>
<span class="p">[</span>
<span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="p">[]),</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">]),</span>
<span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">]),</span>
<span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">])</span>
<span class="p">]</span>
<span class="p">)</span>
<span class="o">@</span><span class="n">cytest</span>
<span class="k">def</span> <span class="nf">test_primes</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">expect</span><span class="p">):</span>
<span class="k">assert</span> <span class="n">primes</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="p">[]</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>pytest
<span class="o">=======================</span> <span class="nb">test </span>session starts <span class="o">=======================</span>
test_foo.py ....
<span class="o">=======================</span> 4 passed <span class="k">in </span>0.16s <span class="o">=========================</span>
</code></pre></div></div>
<h2 id="summary">Summary</h2>
<p>To summarize, it is possible to use <code class="language-plaintext highlighter-rouge">pytest</code> to test Cython code,
using the following approach:</p>
<ol>
<li>
<p>Write the tests in a Cython module (e.g., <code class="language-plaintext highlighter-rouge">foo_tests.pyx</code>),
wrapping each test function in the <code class="language-plaintext highlighter-rouge">@cytest</code> decorator shown above.</p>
</li>
<li>
<p>Create a Python module (e.g., <code class="language-plaintext highlighter-rouge">test_foo.py</code>) that imports the test
functions in <code class="language-plaintext highlighter-rouge">foo_tests.pyx</code>, setting them as its attributes.
If you have several Cython modules containing tests, it might be
convenient to have a single <code class="language-plaintext highlighter-rouge">test_cython.py</code> module that imports
tests from all of them.</p>
</li>
<li>
<p>Run <code class="language-plaintext highlighter-rouge">pytest</code> as usual.</p>
</li>
</ol>
<hr />
<p><strong>Note</strong>: The above examples use Cython 3.0.
If you are using an older version of Cython, it may be necessary
to set the Cython compiler directive <code class="language-plaintext highlighter-rouge">binding=True</code>.
See <a href="https://cython.readthedocs.io/en/latest/src/userguide/source_files_and_compilation.html#compiler-directives">here</a> for more information.</p>This post is about using pytest for testing cdef functions in Cython.Format-on-save with elpy in Emacs2020-01-18T00:00:00+00:002020-01-18T00:00:00+00:00https://shwina.github.io/emacs-python-format-on-save<p>This post is about configuring <a href="https://elpy.readthedocs.io/en/latest/">Elpy</a>
and Emacs to auto-format Python buffers using <a href="https://github.com/psf/black">Black</a>
for a specific project (i.e., using <a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables.html">Directory Variables</a>).</p>
<p><em>Note: for a while, I used <a href="https://github.com/wasamasa/firestarter">firestarter</a> to accomplish this,
but that approach seemed clunky for what I needed.
I mention it here as another useful tool for run-on-save in Emacs</em></p>
<h2 id="configuring-the-elpy-rpc-virtualenv">Configuring the Elpy RPC virtualenv</h2>
<p>When using Elpy, there are two virtualenvs (or alternatively, conda environments)
involved:</p>
<ul>
<li>The virtualenv containing Python packages needed by your project
(we’ll refer to this as the <em>working environment</em>).</li>
<li>The Elpy <a href="https://elpy.readthedocs.io/en/latest/concepts.html#the-rpc-process">RPC virtualenv</a>, where packages used by Elpy (such as <code class="language-plaintext highlighter-rouge">jedi</code> and <code class="language-plaintext highlighter-rouge">black</code>) are installed.</li>
</ul>
<p>Personally, I find it better practice to just use the working environment
as the RPC environment. This allows me to control the versions of formatters
like <code class="language-plaintext highlighter-rouge">black</code> on a per-project basis. But, it also means having to install
them in each of my working environments.</p>
<div class="language-elisp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">;; init.el</span>
<span class="c1">;; Use the working environment as the RPC environment</span>
<span class="p">(</span><span class="k">setq</span> <span class="nv">elpy-rpc-virtualenv-path</span> <span class="ss">'current</span><span class="p">)</span>
</code></pre></div></div>
<p>If you do this, you <em>must</em> activate the working environment in Emacs
at the start of each session.
You can do this with <code class="language-plaintext highlighter-rouge">M-x pyvenv-activate</code> and pointing to the appropriate directory.</p>
<p>At this point, you should run <code class="language-plaintext highlighter-rouge">M-x elpy-config</code> to make sure
Elpy has found the right virtualenv,
and also is able to find the <code class="language-plaintext highlighter-rouge">black</code> formatter.</p>
<h2 id="create-or-update-dir-localsel">Create (or update) <code class="language-plaintext highlighter-rouge">.dir-locals.el</code></h2>
<p>The <a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables.html"><code class="language-plaintext highlighter-rouge">.dir-locals.el</code></a> file allows you to define variables
local to a directory and its sub-directories.
This makes it possible to specialize your Emacs configuration for a project,
while keeping a more general configuration across projects.</p>
<p>Place the <code class="language-plaintext highlighter-rouge">.dir-locals.el</code> file in your project’s top-level directory,
and add the following entry:</p>
<div class="language-elisp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">((</span><span class="nv">python-mode</span> <span class="o">.</span> <span class="p">((</span><span class="nb">eval</span> <span class="o">.</span> <span class="p">(</span><span class="nv">add-hook</span> <span class="ss">'before-save-hook</span> <span class="nf">#'</span><span class="nv">elpy-black-fix-code</span> <span class="no">nil</span> <span class="no">t</span><span class="p">)))))</span>
</code></pre></div></div>
<h2 id="restart-emacs-and-auto-format-away">Restart Emacs and auto-format away!</h2>
<p>After a restart of Emacs, run <code class="language-plaintext highlighter-rouge">M-x pyvenv-activate</code> again,
and navigate to your project directory.
You may be warned by Emacs about the directory local variable
being possibly <a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Safe-File-Variables.html#Safe-File-Variables">unsafe</a>.
Type <code class="language-plaintext highlighter-rouge">!</code> to silence this warning.</p>
<p>That’s it! Python buffers in this project should now be formatted
when you save them:</p>
<p>Before save:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="n">a</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="o">=</span> <span class="mi">3</span><span class="p">):</span>
<span class="n">z</span><span class="o">=</span> <span class="n">x</span><span class="o">+</span> <span class="n">y</span>
<span class="k">return</span> <span class="n">z</span>
</code></pre></div></div>
<p>After save:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="n">a</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">func</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="o">=</span><span class="mi">3</span><span class="p">):</span>
<span class="n">z</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
<span class="k">return</span> <span class="n">z</span>
</code></pre></div></div>
<p>Note that Elpy will respect the Black configuration provided in
a <code class="language-plaintext highlighter-rouge">pyproject.toml</code> placed in the project root directory.</p>This post is about configuring Elpy and Emacs to auto-format Python buffers using Black for a specific project (i.e., using Directory Variables).Got a nice surprise in my e-mail today2016-01-21T00:00:00+00:002016-01-21T00:00:00+00:00https://shwina.github.io/maintainer-certificate<p>Thanks. I’m very grateful, and
so proud to be a part of this community:</p>
<p><a href=""><img src="https://shwina.github.io/images/maintainer-certificate.png" align="middle" height="500" alt="Maintainer Certificate" /></a></p>Thanks. I’m very grateful, and so proud to be a part of this community:Using PETSc for Python with PyCUDA2015-12-27T00:00:00+00:002015-12-27T00:00:00+00:00https://shwina.github.io/petsc-pycuda<p>This post is about how to use the
<a href="https://bitbucket.org/petsc/petsc4py">petsc4py</a>
and <a href="https://mathema.tician.de/software/pycuda/">PyCUDA</a>
together to write applications that use several GPUs
in parallel.</p>
<p>PETSc supports the use of CUDA GPUs via the
<a href="https://developer.nvidia.com/cusp">CUSP</a> C++ library.
The PETSc provided <code class="language-plaintext highlighter-rouge">VECCUSP</code> and <code class="language-plaintext highlighter-rouge">AIJCUSP</code> classes
are used to store vectors and matrices respectively on GPUs.
Vector operations and matrix vector products
with these classes are performed on the GPU.
All the PETSc linear solvers (except BiCG)
are thus able to run entirely on the GPU.</p>
<p>Sometimes, you’ll want to perform computations with vectors
that can’t be expressed using the available vector operations
or matrix-vector products.
In this case, you’ll need access to the
underlying GPU (device) buffer,
which you pass to your own CUDA kernel that performs the computation.</p>
<p>In C++, this is done using the PETSc functions
<code class="language-plaintext highlighter-rouge">VecCUSPGetArrayRead</code> and <code class="language-plaintext highlighter-rouge">VecCUSPGetArrayWrite</code>
that expose the underlying CUSP vectors.
From the CUSP vector,
you can get a pointer to the raw device memory
using the <code class="language-plaintext highlighter-rouge">thrust::raw_pointer_cast</code> function.
This device pointer can be passed to your own custom kernels,
functions from other GPU libraries, whatever.</p>
<p>Here’s how to do it with <code class="language-plaintext highlighter-rouge">petsc4py</code> and PyCUDA.
We’ll multiply two vectors (elementwise)
using a custom kernel.
The following bits of code all go in a single Python script <code class="language-plaintext highlighter-rouge">gpumult.py</code>.
First, we’ll create the input and output vectors:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">petsc4py</span> <span class="kn">import</span> <span class="n">PETSc</span>
<span class="kn">from</span> <span class="nn">pycuda</span> <span class="kn">import</span> <span class="n">autoinit</span>
<span class="kn">import</span> <span class="nn">pycuda.driver</span> <span class="k">as</span> <span class="n">cuda</span>
<span class="kn">import</span> <span class="nn">pycuda.compiler</span> <span class="k">as</span> <span class="n">compiler</span>
<span class="kn">import</span> <span class="nn">pycuda.gpuarray</span> <span class="k">as</span> <span class="n">gpuarray</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">8</span>
<span class="c1"># create input vectors
</span><span class="n">a</span> <span class="o">=</span> <span class="n">PETSc</span><span class="p">.</span><span class="n">Vec</span><span class="p">().</span><span class="n">create</span><span class="p">()</span>
<span class="n">a</span><span class="p">.</span><span class="n">setType</span><span class="p">(</span><span class="s">'cusp'</span><span class="p">)</span>
<span class="n">a</span><span class="p">.</span><span class="n">setSizes</span><span class="p">(</span><span class="n">N</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">PETSc</span><span class="p">.</span><span class="n">Vec</span><span class="p">().</span><span class="n">create</span><span class="p">()</span>
<span class="n">b</span><span class="p">.</span><span class="n">setType</span><span class="p">(</span><span class="s">'cusp'</span><span class="p">)</span>
<span class="n">b</span><span class="p">.</span><span class="n">setSizes</span><span class="p">(</span><span class="n">N</span><span class="p">)</span>
<span class="c1"># create output vectors
</span><span class="n">c</span> <span class="o">=</span> <span class="n">PETSc</span><span class="p">.</span><span class="n">Vec</span><span class="p">().</span><span class="n">create</span><span class="p">()</span>
<span class="n">c</span><span class="p">.</span><span class="n">setType</span><span class="p">(</span><span class="s">'cusp'</span><span class="p">)</span>
<span class="n">c</span><span class="p">.</span><span class="n">setSizes</span><span class="p">(</span><span class="n">N</span><span class="p">)</span>
<span class="c1"># set initial values:
</span><span class="n">a</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="mf">3.0</span><span class="p">)</span>
<span class="n">b</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="mf">2.0</span><span class="p">)</span>
<span class="n">c</span><span class="p">.</span><span class="nb">set</span><span class="p">(</span><span class="mf">0.0</span><span class="p">)</span>
</code></pre></div></div>
<p>Next, we’ll use the <code class="language-plaintext highlighter-rouge">getCUDAHandle</code> method
to get the raw CUDA pointers
of the underlying GPU buffers:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">a_ptr</span> <span class="o">=</span> <span class="n">a</span><span class="p">.</span><span class="n">getCUDAHandle</span><span class="p">()</span>
<span class="n">b_ptr</span> <span class="o">=</span> <span class="n">b</span><span class="p">.</span><span class="n">getCUDAHandle</span><span class="p">()</span>
<span class="n">c_ptr</span> <span class="o">=</span> <span class="n">c</span><span class="p">.</span><span class="n">getCUDAHandle</span><span class="p">()</span>
</code></pre></div></div>
<p>Next, we’ll write a CUDA kernel implementing
the elementwise product, and use PyCUDA to interface with it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">kernel_text</span> <span class="o">=</span> <span class="s">'''
__global__ void gpuElemwiseProduct(double* a_d,
double* b_d, double* c_d, int n) {
int i = threadIdx.x + blockIdx.x*blockDim.x;
c_d[i] = a_d[i]*b_d[i];
}
'''</span>
<span class="n">mult</span> <span class="o">=</span> <span class="n">compiler</span><span class="p">.</span><span class="n">SourceModule</span><span class="p">(</span><span class="n">kernel_text</span><span class="p">,</span>
<span class="n">options</span><span class="o">=</span><span class="p">[</span><span class="s">'-O2'</span><span class="p">]).</span><span class="n">get_function</span><span class="p">(</span><span class="s">'gpuElemwiseProduct'</span><span class="p">)</span>
<span class="n">mult</span><span class="p">.</span><span class="n">prepare</span><span class="p">(</span><span class="s">'PPPi'</span><span class="p">)</span>
</code></pre></div></div>
<p>Now, we’ll perform the multiplication:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mult</span><span class="p">.</span><span class="n">prepared_call</span><span class="p">((</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span> <span class="p">(</span><span class="n">N</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
<span class="n">a_ptr</span><span class="p">,</span> <span class="n">b_ptr</span><span class="p">,</span> <span class="n">c_ptr</span><span class="p">,</span> <span class="n">N</span><span class="p">)</span>
</code></pre></div></div>
<p>The API requires us to “restore” the CUDA handles.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">a</span><span class="p">.</span><span class="n">restoreCUDAHandle</span><span class="p">(</span><span class="n">a_ptr</span><span class="p">)</span>
<span class="n">b</span><span class="p">.</span><span class="n">restoreCUDAHandle</span><span class="p">(</span><span class="n">b_ptr</span><span class="p">)</span>
<span class="n">c</span><span class="p">.</span><span class="n">restoreCUDAHandle</span><span class="p">(</span><span class="n">c_ptr</span><span class="p">)</span>
</code></pre></div></div>
<p>Actually, it doesn’t matter what the input is to
the <code class="language-plaintext highlighter-rouge">restoreCUDAHandle</code> function - but whatever.</p>
<p>Now look at the output vector:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>c.view()
</code></pre></div></div>
<p>Here’s a run of the above program on 2 processes:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mpiexec <span class="nt">-n</span> 2 python gpumult.py
Vec Object: 2 MPI processes
<span class="nb">type</span>: mpicusp
Process <span class="o">[</span>0]
6.
6.
6.
6.
Process <span class="o">[</span>1]
6.
6.
6.
6.
</code></pre></div></div>
<p>You can also use the
raw pointer to construct a PyCUDA GPUArray,
or pass it to a function from the
<code class="language-plaintext highlighter-rouge">scikits.cuda</code> library, or wherever else.</p>This post is about how to use the petsc4py and PyCUDA together to write applications that use several GPUs in parallel. PETSc supports the use of CUDA GPUs via the CUSP C++ library. The PETSc provided VECCUSP and AIJCUSP classes are used to store vectors and matrices respectively on GPUs. Vector operations and matrix vector products with these classes are performed on the GPU. All the PETSc linear solvers (except BiCG) are thus able to run entirely on the GPU. Sometimes, you’ll want to perform computations with vectors that can’t be expressed using the available vector operations or matrix-vector products. In this case, you’ll need access to the underlying GPU (device) buffer, which you pass to your own CUDA kernel that performs the computation. In C++, this is done using the PETSc functions VecCUSPGetArrayRead and VecCUSPGetArrayWrite that expose the underlying CUSP vectors. From the CUSP vector, you can get a pointer to the raw device memory using the thrust::raw_pointer_cast function. This device pointer can be passed to your own custom kernels, functions from other GPU libraries, whatever. Here’s how to do it with petsc4py and PyCUDA. We’ll multiply two vectors (elementwise) using a custom kernel. The following bits of code all go in a single Python script gpumult.py. First, we’ll create the input and output vectors: from petsc4py import PETSc from pycuda import autoinit import pycuda.driver as cuda import pycuda.compiler as compiler import pycuda.gpuarray as gpuarray import numpy as np N = 8 # create input vectors a = PETSc.Vec().create() a.setType('cusp') a.setSizes(N) b = PETSc.Vec().create() b.setType('cusp') b.setSizes(N) # create output vectors c = PETSc.Vec().create() c.setType('cusp') c.setSizes(N) # set initial values: a.set(3.0) b.set(2.0) c.set(0.0) Next, we’ll use the getCUDAHandle method to get the raw CUDA pointers of the underlying GPU buffers: a_ptr = a.getCUDAHandle() b_ptr = b.getCUDAHandle() c_ptr = c.getCUDAHandle() Next, we’ll write a CUDA kernel implementing the elementwise product, and use PyCUDA to interface with it: kernel_text = ''' __global__ void gpuElemwiseProduct(double* a_d, double* b_d, double* c_d, int n) { int i = threadIdx.x + blockIdx.x*blockDim.x; c_d[i] = a_d[i]*b_d[i]; } ''' mult = compiler.SourceModule(kernel_text, options=['-O2']).get_function('gpuElemwiseProduct') mult.prepare('PPPi') Now, we’ll perform the multiplication: mult.prepared_call((1, 1, 1), (N, 1, 1), a_ptr, b_ptr, c_ptr, N) The API requires us to “restore” the CUDA handles. a.restoreCUDAHandle(a_ptr) b.restoreCUDAHandle(b_ptr) c.restoreCUDAHandle(c_ptr) Actually, it doesn’t matter what the input is to the restoreCUDAHandle function - but whatever. Now look at the output vector: c.view() Here’s a run of the above program on 2 processes: $ mpiexec -n 2 python gpumult.py Vec Object: 2 MPI processes type: mpicusp Process [0] 6. 6. 6. 6. Process [1] 6. 6. 6. 6. You can also use the raw pointer to construct a PyCUDA GPUArray, or pass it to a function from the scikits.cuda library, or wherever else.Wildcard completion in IPython2014-11-10T00:00:00+00:002014-11-10T00:00:00+00:00https://shwina.github.io/ipython-wildcard<p>I wanted to look up
all the top-level, grid related functions
available in NumPy.
On the IPython shell,
on a whim,
I typed:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import numpy as np
np.*grid*?
</code></pre></div></div>
<p>not knowing what to expect.
It did just what I wanted:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>np.meshgrid
np.mgrid
np.ogrid
</code></pre></div></div>
<p>That made my day.
Thanks, IPython!</p>I wanted to look up all the top-level, grid related functions available in NumPy. On the IPython shell, on a whim, I typed: import numpy as np np.*grid*? not knowing what to expect. It did just what I wanted: np.meshgrid np.mgrid np.ogrid That made my day. Thanks, IPython!Profiling Parallel Programs2014-11-05T00:00:00+00:002014-11-05T00:00:00+00:00https://shwina.github.io/profiling-parallel<p>In high performance computing,
performance is kind of a big deal.
And the first step in performance analysis
and performance <em>improvement</em>
is
<a href="https://en.wikipedia.org/wiki/Profiling_%28computer_programming%29">profiling</a>.</p>
<p>High performance computing almost always entails some
form of <a href="https://en.wikipedia.org/wiki/Parallel_computing">parallelism</a>.
And parallel programs are plain hard. They’re harder to write,
harder to debug, and harder to profile.</p>
<h2 id="gprof">gprof</h2>
<p><code class="language-plaintext highlighter-rouge">gprof</code> is pretty great. Just compile your code with <code class="language-plaintext highlighter-rouge">-pg</code>, and <code class="language-plaintext highlighter-rouge">-g</code>,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gcc <span class="nt">-pg</span> <span class="nt">-g</span> <span class="nt">-O0</span> hello.c bye.c <span class="nt">-o</span> hibye.exe
</code></pre></div></div>
<p>run your code as usual,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./hibye.exe
</code></pre></div></div>
<p>and you’ll see <code class="language-plaintext highlighter-rouge">gmon.out</code>. Now,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ gprof hibye.exe gmon.out
</code></pre></div></div>
<p>should summarize the performance of your code.
Beware, <code class="language-plaintext highlighter-rouge">gprof</code> will not
pick up on any calls to shared library functions.
OK, that’s a downer, and
there’s lots more. But it’s easy to use, and gives me quick results.
With the legacy code I work with, where there <em>are</em> no shared library calls,
<code class="language-plaintext highlighter-rouge">gprof</code> is pretty awesome.</p>
<h2 id="gprof--mpi">gprof + MPI</h2>
<p><code class="language-plaintext highlighter-rouge">gprof</code> isn’t designed to work with MPI code.
But, as is generally the case with these things,
it’s possible with sufficient abuse:</p>
<p>First, set the environment variable <code class="language-plaintext highlighter-rouge">GMON_OUT_PREFIX</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">export </span><span class="nv">GMON_OUT_PREFIX</span><span class="o">=</span>gmon.out-
</code></pre></div></div>
<p>Then, the usual business:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mpicc <span class="nt">-pg</span> <span class="nt">-g</span> <span class="nt">-O0</span> hello.c bye.c <span class="nt">-o</span> hibye.exe
<span class="nv">$ </span>mpiexec <span class="nt">-n</span> 32 hibye.exe
</code></pre></div></div>
<p>You should see 32 (or however many processes) files,
with names <code class="language-plaintext highlighter-rouge">gmon.out-<pid></code>.
This is an undocumented feature of <code class="language-plaintext highlighter-rouge">glibc</code>,
and it really shouldn’t be - it’s massively useful.</p>
<p>Now you have a separate <code class="language-plaintext highlighter-rouge">gmon.out</code> file for every
MPI process. Awesome. Sum them:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gprof <span class="nt">-s</span> hibye.exe gmon.out-<span class="k">*</span>
</code></pre></div></div>
<p>And use the resulting <code class="language-plaintext highlighter-rouge">gmon.sum</code> to generate
<code class="language-plaintext highlighter-rouge">gprof</code> output:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gprof hibye.exe gmon.sum
</code></pre></div></div>
<p>Now, I haven’t figured out how to replace the <code class="language-plaintext highlighter-rouge">pid</code>
with the MPI rank -
this could be exponentially more useful to some users.
And the method mentioned in the source doesn’t really
seem to be working.
But I’m sure this is possible with some ingenuity.</p>
<h2 id="mpip">mpiP</h2>
<p><a href="https://github.com/LLNL/mpiP">mpiP</a> is a neat little
tool for profiling MPI applications.
In particular, it’s extremely useful in figuring out
how much your application is spending time <em>communicating</em>
relative to <em>computing</em>.</p>
<p>The documentation for setting up and using <code class="language-plaintext highlighter-rouge">mpiP</code>
is complete (good), but small (better).
Once you have <code class="language-plaintext highlighter-rouge">mpiP</code> set up, profiling your code is
as easy as linking it with the <code class="language-plaintext highlighter-rouge">mpiP</code> library and some
other stuff it needs:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mpicc <span class="nt">-g</span> <span class="nt">-O0</span> hello.c bye.c <span class="nt">-o</span> hibye.exe <span class="nt">-lmpiP</span> <span class="nt">-liberty</span> <span class="nt">-lbfd</span> <span class="nt">-lunwind</span>
</code></pre></div></div>
<p>Running your code (<code class="language-plaintext highlighter-rouge">mpiexec</code>) will produce <code class="language-plaintext highlighter-rouge">mpiP</code> output.</p>
<p>I’ve found that while <code class="language-plaintext highlighter-rouge">gprof</code> and <code class="language-plaintext highlighter-rouge">mpiP</code> are great tools
that do different things, using them <em>both</em> gives
me a very good idea of where my programs are spending time
and where I should focus optimization efforts.</p>In high performance computing, performance is kind of a big deal. And the first step in performance analysis and performance improvement is profiling. High performance computing almost always entails some form of parallelism. And parallel programs are plain hard. They’re harder to write, harder to debug, and harder to profile. gprof gprof is pretty great. Just compile your code with -pg, and -g, $ gcc -pg -g -O0 hello.c bye.c -o hibye.exe run your code as usual, $ ./hibye.exe and you’ll see gmon.out. Now, $ gprof hibye.exe gmon.out should summarize the performance of your code. Beware, gprof will not pick up on any calls to shared library functions. OK, that’s a downer, and there’s lots more. But it’s easy to use, and gives me quick results. With the legacy code I work with, where there are no shared library calls, gprof is pretty awesome. gprof + MPI gprof isn’t designed to work with MPI code. But, as is generally the case with these things, it’s possible with sufficient abuse: First, set the environment variable GMON_OUT_PREFIX: $ export GMON_OUT_PREFIX=gmon.out- Then, the usual business: $ mpicc -pg -g -O0 hello.c bye.c -o hibye.exe $ mpiexec -n 32 hibye.exe You should see 32 (or however many processes) files, with names gmon.out-<pid>. This is an undocumented feature of glibc, and it really shouldn’t be - it’s massively useful. Now you have a separate gmon.out file for every MPI process. Awesome. Sum them: $ gprof -s hibye.exe gmon.out-* And use the resulting gmon.sum to generate gprof output: $ gprof hibye.exe gmon.sum Now, I haven’t figured out how to replace the pid with the MPI rank - this could be exponentially more useful to some users. And the method mentioned in the source doesn’t really seem to be working. But I’m sure this is possible with some ingenuity. mpiP mpiP is a neat little tool for profiling MPI applications. In particular, it’s extremely useful in figuring out how much your application is spending time communicating relative to computing. The documentation for setting up and using mpiP is complete (good), but small (better). Once you have mpiP set up, profiling your code is as easy as linking it with the mpiP library and some other stuff it needs: $ mpicc -g -O0 hello.c bye.c -o hibye.exe -lmpiP -liberty -lbfd -lunwind Running your code (mpiexec) will produce mpiP output. I’ve found that while gprof and mpiP are great tools that do different things, using them both gives me a very good idea of where my programs are spending time and where I should focus optimization efforts.Teaching MATLAB2014-09-14T00:00:00+00:002014-09-14T00:00:00+00:00https://shwina.github.io/teaching-matlab<p>I had the opportunity to introduce MATLAB to two groups
of 30 graduate students fitting the following profile:</p>
<ol>
<li>
<p>Non-majors in Computer Science</p>
</li>
<li>
<p>Novice MATLAB programmers</p>
</li>
<li>
<p>Programming experience in <em>some</em> language (C/Java)</p>
</li>
</ol>
<p>Learners came in
with a broad range of expectations:</p>
<ul>
<li>What is MATLAB, and what are people doing with it?,</li>
<li>How to do <em>X</em> with MATLAB?,</li>
<li>How can I use MATLAB more effectively than I already am?</li>
</ul>
<p>I’m a huge fan of <a href="https://software-carpentry.org/" target="_blank">
Software Carpentry</a>
and their <em>evidence-based</em>
approach to teaching. The
argument makes sense: we are scientists, and founding our teaching
methods on hard research–or at the very least, <em>some</em> data–is likely a
better idea than using arbitrary lesson plans and teaching techniques.
Accordingly, as a starting point, I chose the Software Carpentry
<a href="https://swcarpentry.github.io/python-novice-inflammation/" target="_blank">
lesson material on Python </a>
These lessons have the immense value of <em>feedback</em> from several
workshops that have been run by trained instructors all over the world.
Of course, the first step would have had to be <em>translating</em> all that lesson
material from Python to MATLAB.
Which <a href="https://github.com/ashwinsrnth/bc/tree/master/novice/matlab" target="_blank">
I did</a>. This
doubled up as my very first contribution to the open-source community (yay).</p>
<p>The concepts covered are, in this order:</p>
<ul>
<li>Loading, analyzing and visualizing data from a file</li>
<li>Writing MATLAB scripts and loops</li>
<li>How and why to write MATLAB functions</li>
<li>Conditionals in MATLAB</li>
<li>Writing tests for functions</li>
</ul>
<p>The lessons are built around a central task: analyzing and visualizing
the data from several .csv files. With every lesson, we make our code
for doing this a little better. For example, in lesson 1, we load,
analyze and visualize the data interactively (from the command line).
In lesson 2, we put those commands in a script, and discuss the
pros and cons over computing interactively. We proceed to introduce
loops, and modify our script to analyze <em>several</em> datasets. And so on.</p>
<p>I’ve done sessions on MATLAB before, and I used to follow a “textbook”
approach: exposing ideas in the same order that they would appear in
a textbook on MATLAB, for example:</p>
<ul>
<li>Variables and Statements</li>
<li>Vectors</li>
<li>Plots</li>
<li>Matrices</li>
<li>Loops</li>
<li>Conditionals</li>
<li>Scripts</li>
<li>Functions</li>
</ul>
<p>So, in one of my earlier sessions, the first few lines of code we would
type in to the command line would be something like this:</p>
<div class="language-matlab highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>></span> <span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="o">>></span> <span class="n">b</span> <span class="o">=</span> <span class="mi">2</span>
<span class="o">>></span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="o">>></span> <span class="n">a</span> <span class="o">*</span> <span class="n">b</span>
<span class="o">>></span> <span class="n">c</span> <span class="o">=</span> <span class="p">[</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">]</span>
</code></pre></div></div>
<p>Compare <em>that</em> to the first couple of lines of code that we type in now:</p>
<div class="language-matlab highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>></span> <span class="n">patient_data</span> <span class="o">=</span> <span class="nb">csvread</span><span class="p">(</span><span class="err">`</span><span class="n">inflammation</span><span class="o">-</span><span class="mf">01.</span><span class="n">csv</span><span class="err">`</span><span class="p">);</span>
<span class="o">>></span> <span class="nb">imagesc</span><span class="p">(</span><span class="n">patient_data</span><span class="p">)</span>
</code></pre></div></div>
<p>I think that exposing this sort of powerful functionality early is important:
it makes learners feel like “this might actually be worth my time” and
encourages them to participate more.</p>
<p>Getting novice programmers to follow along command-by-command is slow: they’re
going to meet with a lot of errors, even with the simplest of commands. The most common
mistakes I’ve seen learners make in workshops:</p>
<ul>
<li>Typos</li>
<li>Calling scripts/functions from the wrong directory</li>
</ul>
<p>This is natural and expected; a new programming environment takes time to get
used to, and learners simply don’t have enough context to make sense of
error messages. It’s tempting then, to demo-ize the whole thing and keep
participants from writing too much code. Of course, that’s a bad idea, and I prefer an
approach that’s somewhere in-between:</p>
<p><strong>Commands</strong></p>
<p>Have learners type out commands on the shell while introducing ideas
for the first time and <em>demonstrate</em> commands when expanding on them
or explaining subtleties.</p>
<p><strong>Scripts/functions</strong></p>
<p>Have learners type out stripped-down, simple versions of more complex
scripts. For example, instead of having
learners write a script that loops
over several datasets, performs analyses, and plots various figures for
each, have them type out and execute a script that performs a single analysis
on a single dataset, and produces a single figure. <em>Then</em> ask them to
look at a more complex version of that script that was distributed to them
at the beginning of the session.</p>
<p>I also experimented with a workshop etherpad, and gave learners the option
of taking notes there instead of on their personal notepads/computers.
Most learners preferred not to interact too much with the etherpad,
I’m not sure why - maybe this should be part of the feedback - but here
are some possible reasons:</p>
<ul>
<li>Not enough time</li>
<li>Not familiar with etherpad</li>
<li>Not as convenient/useful as personal notes</li>
</ul>
<p>I gave participants the option of providing feedback, either on the public
etherpad, or on pieces of paper. Feedback on paper was generally more
specific and comprehensive. Response was positive, with complaints
about not having enough time and not covering enough/specific material.
The structure and content of the lessons was generally appreciated,
although there were mixed opinions on the section on testing.</p>I had the opportunity to introduce MATLAB to two groups of 30 graduate students fitting the following profile: Non-majors in Computer Science Novice MATLAB programmers Programming experience in some language (C/Java) Learners came in with a broad range of expectations: What is MATLAB, and what are people doing with it?, How to do X with MATLAB?, How can I use MATLAB more effectively than I already am? I’m a huge fan of Software Carpentry and their evidence-based approach to teaching. The argument makes sense: we are scientists, and founding our teaching methods on hard research–or at the very least, some data–is likely a better idea than using arbitrary lesson plans and teaching techniques. Accordingly, as a starting point, I chose the Software Carpentry lesson material on Python These lessons have the immense value of feedback from several workshops that have been run by trained instructors all over the world. Of course, the first step would have had to be translating all that lesson material from Python to MATLAB. Which I did. This doubled up as my very first contribution to the open-source community (yay). The concepts covered are, in this order: Loading, analyzing and visualizing data from a file Writing MATLAB scripts and loops How and why to write MATLAB functions Conditionals in MATLAB Writing tests for functions The lessons are built around a central task: analyzing and visualizing the data from several .csv files. With every lesson, we make our code for doing this a little better. For example, in lesson 1, we load, analyze and visualize the data interactively (from the command line). In lesson 2, we put those commands in a script, and discuss the pros and cons over computing interactively. We proceed to introduce loops, and modify our script to analyze several datasets. And so on. I’ve done sessions on MATLAB before, and I used to follow a “textbook” approach: exposing ideas in the same order that they would appear in a textbook on MATLAB, for example: Variables and Statements Vectors Plots Matrices Loops Conditionals Scripts Functions So, in one of my earlier sessions, the first few lines of code we would type in to the command line would be something like this: >> a = 1 >> b = 2 >> a + b >> a * b >> c = [a, b]