Dan AloniDan AloniZola2023-12-16T00:00:00+00:00https://blog.aloni.org/atom.xmlRegret Minimization Test2023-12-16T00:00:00+00:002023-12-16T00:00:00+00:00https://blog.aloni.org/posts/regret-minimization-test/<p>You would save yourself pain if you have the least amount of regrets.</p>
<p>After accumulating a life of decisions, it is important to know when to look back. I can test a decision I've made in the past for whether I have a real regret or not.</p>
<p>The test goes like this:</p>
<blockquote>
<p>If hypothetically, I would unequivocally want to send myself a note from the future, straight to the decision time, on which only the text "don't do it!" is written <em>and nothing else</em>, then it is a real regret.</p>
</blockquote>
<p>Anything that does not meet this criteria is a fake regret. Any decision that needs more information than what you had at that time — cannot be really regretted.</p>
<p>This approach to defining and testing regret seems logical and practical. It filters out scenarios where the regret is based on information or perspectives you couldn't have had at the time. It focuses on decisions where, even with the limited info you had then, a simple warning from the future would have been enough to change your mind. This method acknowledges that hindsight is clearer, but doesn't let it unfairly influence your judgment of past decisions.</p>
<p>You can regard this test a complementary tool to the "regret minimization framework" by Jeff Bezos, but which works only after the fact.</p>
Bash #4 - Bash Docstrings2023-07-31T00:00:00+00:002023-07-31T00:00:00+00:00https://blog.aloni.org/posts/bash-docstrings/<p>Previous post: <a href="../bash-path-wrapping-executables">#3</a>.</p>
<p>Once of Python's nice features are <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://peps.python.org/pep-0257/">docstrings<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, for which a <code>help</code> function on a module generates a nice documentation screen. Can we follow a similar technique in our <code>bash</code>? Yes we can!</p>
<p>We'll do this as an extension of the <a href="../bash-functional-command-relay">functional command
relay</a> technique.</p>
<span id="continue-reading"></span>
<p>Adding docstrings can be done by inserting special inner functions that don't
execute on command execution. For example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source shell bash"><span class="meta function shell"><span class="entity name function shell">build</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="meta function shell"> <span class="entity name function shell">__docstring__</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="punctuation definition string end shell">"</span></span><span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><params>
Build and work on stuff
-j <core> - number of cores
<span class="punctuation definition string end shell">"</span></span><span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="punctuation definition string end shell">"</span></span></span>
<span class="punctuation section braces end shell">}</span></span>
<span class="meta function-call shell"><span class="support function colon shell">:</span></span> <span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> << your build implementation here >></span><span class="comment line number-sign shell">
</span><span class="punctuation section braces end shell">}</span></span>
<span class="meta function shell"><span class="entity name function shell">run</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="meta function shell"> <span class="entity name function shell">__docstring__</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="punctuation definition string end shell">"</span></span><span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><params>
Run stuff
<span class="punctuation definition string end shell">"</span></span><span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="punctuation definition string end shell">"</span></span></span>
<span class="punctuation section braces end shell">}</span></span>
<span class="meta function-call shell"><span class="support function colon shell">:</span></span> <span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> << your run implementation here >></span><span class="comment line number-sign shell">
</span><span class="punctuation section braces end shell">}</span></span>
</span></td></pre></tr></tbody></table></div>
<p>We can pick this up in a new <code>help</code> command. Suppose we define <code>help</code> as such:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source shell bash"><span class="meta function shell"><span class="entity name function shell">help</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="meta function shell"> <span class="entity name function shell">doc-iter-functions</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="meta function-call shell"> <span class="storage modifier shell">local</span> <span class="variable other readwrite assignment shell">funcname</span></span>
<span class="keyword control loop for shell">for</span><span class="meta group for shell"> funcname <span class="keyword control in shell">in</span> <span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="storage modifier shell">declare</span> <span class="variable parameter option shell"><span class="punctuation definition parameter shell">-</span>F</span> </span><span class="meta function-call shell"></span><span class="keyword operator logical pipe shell">|</span> <span class="meta function-call shell"><span class="variable function shell">grep</span></span><span class="meta function-call arguments shell"> <span class="punctuation separator continuation line shell">\
</span><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>Ev</span> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>^declare -fx<span class="punctuation definition string end shell">'</span></span></span> <span class="keyword operator logical pipe shell">|</span> <span class="meta function-call shell"><span class="variable function shell">sed</span></span><span class="meta function-call arguments shell"> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>s/declare -f //g<span class="punctuation definition string end shell">'</span></span></span> <span class="keyword operator logical pipe shell">|</span> <span class="meta function-call shell"><span class="variable function shell">sort</span></span><span class="punctuation section parens end shell">)</span></span></span><span class="keyword operator logical continue shell">;</span>
<span class="keyword control loop do shell">do</span>
<span class="meta function shell"> <span class="storage type function shell">function</span> <span class="entity name function shell">__docstring__</span> <span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span> <span class="meta function-call shell"><span class="support function colon shell">:</span></span><span class="keyword operator logical continue shell">;</span> <span class="punctuation section braces end shell">}</span></span>
<span class="meta function-call shell"><span class="support function eval shell">eval</span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="support function type shell">type</span></span><span class="meta function-call arguments shell"> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section expansion parameter begin shell">{</span></span><span class="meta group expansion parameter shell"><span class="variable other readwrite shell">funcname</span></span><span class="meta group expansion parameter shell"><span class="punctuation section expansion parameter end shell">}</span></span></span> <span class="keyword operator logical pipe shell">|</span> <span class="meta function-call shell"><span class="variable function shell">awk</span></span><span class="meta function-call arguments shell"> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>/function __docstring__ \(\)/,/};/<span class="punctuation definition string end shell">'</span></span></span><span class="punctuation section parens end shell">)</span></span><span class="punctuation definition string end shell">"</span></span></span>
<span class="keyword control conditional if shell">if</span> <span class="meta function-call arguments shell"><span class="support function double-brace begin shell">[[</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="variable function shell">__docstring__</span></span><span class="punctuation section parens end shell">)</span></span><span class="punctuation definition string end shell">"</span></span> <span class="keyword operator logical shell">!=</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="punctuation definition string end shell">"</span></span> <span class="support function double-brace end shell">]]</span></span> <span class="keyword operator logical continue shell">;</span> <span class="keyword control conditional then shell">then</span>
<span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="variable parameter option shell"><span class="punctuation definition parameter shell">-</span>n</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span> script <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section expansion parameter begin shell">{</span></span><span class="meta group expansion parameter shell"><span class="variable other readwrite shell">funcname</span></span><span class="meta group expansion parameter shell"><span class="punctuation section expansion parameter end shell">}</span></span><span class="punctuation definition string end shell">"</span></span></span>
<span class="meta function-call shell"><span class="variable function shell">__docstring__</span></span>
<span class="meta function-call shell"><span class="support function echo shell">echo</span></span>
<span class="keyword control conditional end shell">fi</span>
<span class="keyword control loop end shell">done</span>
<span class="punctuation section braces end shell">}</span></span>
<span class="meta function-call shell"><span class="variable function shell">cat</span></span><span class="meta function-call arguments shell"> <span class="string unquoted heredoc shell"><span class="keyword operator assignment redirection shell"><<</span> <span class="keyword control heredoc-token shell">EOF</span></span><span class="string unquoted heredoc shell">
Help screen
<span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="variable function shell">doc-iter-functions</span></span><span class="punctuation section parens end shell">)</span></span>
<span class="keyword control heredoc-token shell">EOF</span></span></span>
<span class="punctuation section braces end shell">}</span></span>
</span></td></pre></tr></tbody></table></div>
<p>Our script now has a help screen! We can view it by executing the <code>help</code> command:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ ./script help
Help screen
script build <params>
Build and work on stuff
-j <core> - number of cores
script run <params>
Run stuff
</span></td></pre></tr></tbody></table></div>
<p>If you are not sure how these bash function translate to commands to the script, read the previous post regarding <a href="../bash-functional-command-relay">functional command relay</a>.</p>
Putting Tests in a Separate Git Repo2022-06-02T00:00:00+00:002022-06-02T00:00:00+00:00https://blog.aloni.org/posts/putting-tests-in-a-separate-git-repo/<p>In the debate on whether to put everything in the so-called mono-repo, system
tests are usually not considered a separate codebase. My take on the mono-repo
vs multiple-repo debate is "split histories on a stable API boundary", and I
think that for system tests this split becomes valid too.</p>
<p>With the emerging CI/CD standard stacks that encourage us to put the tests with
the code being tested, it seems to me that the tendency to split the tests from
the main code gets even weaker, where it promotes a defaulted mono-repo design
that unites main code and test code together to the same repository.</p>
<p>In this post I present arguments mostly against putting system tests in the
same Git repository.</p>
<h2 id="api-compatibility">API compatibility</h2>
<p>If a system presents APIs to external entities (i.e. your customers), it means
that your system tests should be able to do feature testing using these APIs in
a way that would allow new tests to work on old code. Tests relying on features
not supported by the old code can be auto-disabled.</p>
<p>The system tests are just another customer that insists on not upgrading across
this stable API boundary. The high-level tests are aware of this API layer, and
therefore can be versioned separately.</p>
<h2 id="git-bisect">Git Bisect</h2>
<p>If you are using Git bisect to pinpoint the change that introduced a bug, then
the test code that reproduces it may be quite new, and the Git history in the
older versions where the bug was introduces does not include that test code at
all. Therefore, it is only natural that the test lives in a separate repo,
otherwise you would need to do weird <code>git checkout</code> maneuvers to get the testee
and tester working on the same run.</p>
<h2 id="exception-tightly-integrated-unit-tests">Exception: tightly integrated unit tests</h2>
<p>One may go too far saying let's put all tests in a separate repo. However, I
think that line should be drawn carefully and checked for whether it works in
practice. For the tests that are tightly bound to the code being tested, for
instance unit tests that depend on internal interfaces or symbols, it may be
harder to apply this principle.</p>
Proper use of Git tags2022-05-23T00:00:00+00:002022-05-23T00:00:00+00:00https://blog.aloni.org/posts/proper-use-of-git-tags/<p>In the Linux kernel project, the first Git-managed project in existence, due to
its <i>proper</i> use of <a href="https://blog.aloni.org/posts/proper-use-of-git-tags/28cf22d0ba28">Git tags</a>, the revision <code>v4.11-rc7-87-g28cf22d0ba28</code> is
equivalent to <code>28cf22d0ba28</code>. This is due to existence of the annotated
<code>v4.11-rc7</code> tag for an ancestor commit.</p>
<p>The nicer string above is the output of <code>git describe 28cf22d0ba28</code>. The <code>git describe</code> command can relate any commit to tags in the history. Any command
that accepts the Git hash behaves the same when the output of <code>git describe</code> is
passed instead. It works in the following way: when Git is presented with
<code>[anystring]-g[hash]</code>, it throws away the <code>[anystring]-g</code> prefix and treats the
<code>[hash]</code> part as the desired commit to resolve.</p>
<p>Git tags let us prettify any Git hash with the software version from which it
is derived. However in many projects Git tags are often not used to their full
potential in the best case, and in the worst case they are being misused.</p>
<p>In this post I provide bunch of recommendations to assist in achieving the
nicety that is obtainable from Git tags in any project.</p>
<h2 id="tag-push-permissions">Tag push permissions</h2>
<p>A common pitfall is to let any developer push Git tags. The following scenario
happened to me <strong>in more than one company</strong>:</p>
<ul>
<li>Someone pushed a <code>test</code> tag.</li>
<li>Someone else created <code>test</code> subdir in the project's root.</li>
</ul>
<p>Now, <code>git log test</code> is an ambiguous command that does not know what to do.
Should it show the commits starting from the commit pointed by <code>test</code>, or the
commits from <code>HEAD</code> that modified that <code>test</code> directory? Or maybe the developer
has a local <code>test</code> branch? Headache.</p>
<p>If possible, configure the Git server to accept tag pushes only from release
managers.</p>
<h2 id="tag-naming">Tag naming</h2>
<p>Tags are unique and live forever in the repo.</p>
<p>Pick good tag names that do not cause ambiguity with your branch name
convention.</p>
<p>A nice convention is the following: to mark versions, start with <code>v</code>. That way,
shell completion works nicely upon trying to complete <code>v</code>. If you drop the <code>v</code>
and instead go with the version number immediately .e.g <code>1.</code>, shell completion
will struggle to show completions between tags starting with <code>1</code> and all
commits whose git hashes start with <code>1</code>.</p>
<h2 id="only-used-annotated-tags-not-lightweight-tags">Only used annotated tags, not lightweight tags</h2>
<p>By default <code>git tag</code> creates lightweight tags (non-annotated tags). <strong>Do</strong> pass
the <code>-a</code> switch to create annotated tags. This way, meta-data from the creation
of the tag is registered just like with commits. Also, <code>git describe</code> would
work out of the box and won't need the <code>--tags</code> flag that is needed to force it
to consider lightweight tags too.</p>
<h2 id="don-t-rely-on-tags-alone-for-versioning">Don't rely on tags alone for versioning</h2>
<p>Git tags are just complementary information to Git, they are not the source
itself. Use both Git tags and a source tree file to tell the version of the
code. This is usually the version string in the package manager manifest of
your ecosystem of choice. But how to do that properly? i.e. which commits
should be tagged? This takes us to the next recommendation —</p>
<h2 id="the-commit-that-changed-the-version-in-source-is-the-one-to-be-tagged">The commit that changed the version in source is the one to be tagged</h2>
<p>By convention, the commit that changed the version in the source <strong>is</strong> the one
that should be tagged, not the commit that merged it into the main branch. This
way, <code>git describe</code> shows the expected result.</p>
<h2 id="further-resources">Further resources</h2>
<ul>
<li><span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://git-scm.com/book/en/v2/Git-Basics-Tagging">Git Basics - tagging<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span></li>
<li><span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://medium.com/@keshshen/lets-start-tagging-88c299b6b331">Let's start tagging!<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span></li>
</ul>
The SQLite SET-AND Gotcha2022-02-21T00:00:00+00:002022-02-21T00:00:00+00:00https://blog.aloni.org/posts/the-sqlite-set-and-gotcha/<p>In SQLite, suppose that we have the following table:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source sql"><span class="meta create sql"><span class="keyword other create sql">CREATE</span> <span class="keyword other sql">TABLE</span> </span><span class="meta toc-list full-identifier sql">IF NOT EXISTS </span><span class="meta toc-list full-identifier sql"><span class="entity name function sql">external_data</span></span> (
name <span class="storage type sql">TEXT</span> <span class="storage modifier sql">PRIMARY KEY</span> <span class="keyword operator logical sql">NOT</span> <span class="constant language sql">NULL</span>,
atime <span class="storage type sql">TEXT</span> <span class="keyword operator logical sql">NOT</span> <span class="constant language sql">NULL</span>,
content <span class="storage type sql">TEXT</span> <span class="keyword operator logical sql">NOT</span> <span class="constant language sql">NULL</span>
);
</span></td></pre></tr></tbody></table></div>
<p>And we have inserted a row:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source sql"> <span class="keyword other DML sql">INSERT INTO</span> external_data
<span class="keyword other DML II sql">VALUES</span> (<span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>name<span class="punctuation definition string end sql">"</span></span>, <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>october<span class="punctuation definition string end sql">"</span></span>, <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>a<span class="punctuation definition string end sql">"</span></span>);
</span></td></pre></tr></tbody></table></div>
<p>Plus, we have executed an 'upsert' statement:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source sql"> <span class="keyword other DML sql">INSERT INTO</span> external_data
<span class="keyword other DML II sql">VALUES</span> (<span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>name<span class="punctuation definition string end sql">"</span></span>, <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>december<span class="punctuation definition string end sql">"</span></span>, <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>b<span class="punctuation definition string end sql">"</span></span>)
ON CONFLICT (name)
DO <span class="keyword other DML sql">UPDATE</span> <span class="keyword other DML sql">SET</span> content <span class="keyword operator comparison sql">=</span> <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>b<span class="punctuation definition string end sql">"</span></span>
<span class="keyword operator logical sql">AND</span> atime <span class="keyword operator comparison sql">=</span> <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>december<span class="punctuation definition string end sql">"</span></span>;
</span></td></pre></tr></tbody></table></div>
<p>Without conflicts, it does what you expect.
But we do have a conflict, and we end up with the following in the table:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source sql">sqlite<span class="keyword operator comparison sql">></span> <span class="keyword other DML sql">SELECT</span> <span class="variable language star sql">*</span> <span class="keyword other DML sql">FROM</span> external_data ;
name|october|<span class="constant numeric sql">0</span>
</span></td></pre></tr></tbody></table></div>
<p>Instead of what we wanted to accomplish with the query, <code>content</code> received the
value of <code>"0"</code> while the <code>atime</code> field remained with the old value <code>"october"</code>
instead of being set with <code>"december"</code>.</p>
<h3 id="what-the-hell-is-going-on">What the hell is going on</h3>
<p>Well, we misplaced the <code>SET</code> separator <code>,</code> with <code>AND</code> because having <code>AND</code>
in <code>WHERE</code> makes sense and we casually did cut&paste from a <code>WHERE</code>
statement to a <code>SET</code> statement... well, this <code>SET</code> statement is in fact
equivalent to the following:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source sql"><span class="keyword other DML sql">SET</span> content <span class="keyword operator comparison sql">=</span> (<span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>b<span class="punctuation definition string end sql">"</span></span> <span class="keyword operator logical sql">AND</span> atime <span class="keyword operator comparison sql">=</span> <span class="string quoted double sql"><span class="punctuation definition string begin sql">"</span>december<span class="punctuation definition string end sql">"</span></span>)
</span></td></pre></tr></tbody></table></div>
<p>Where <code>("b" AND atime = "december")</code> evaluates to 0 because the second <code>AND</code>
operand is not true.</p>
<h3 id="wait-but-sqlite-has-strictmode">Wait, but SQLite has 'StrictMode'</h3>
<p>Yes, there's <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://sqlite.org/src/wiki?name=StrictMode">StrictMode<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, but it
doesn't help here as of SQLite 3.37.2. A String is a valid operand to <code>AND</code>.</p>
<h3 id="wait-again-but-there-s-strict-for-create-table">Wait again, but there's <code>STRICT</code> for <code>CREATE TABLE</code></h3>
<p>Nope, doesn't help either. Sorry.</p>
<p><span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://sqlime.org/#deta:xd5y9mf00fsu">sqlime playground<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
Bash #3 - PATH-wrapping executables2022-02-18T00:00:00+00:002022-02-18T00:00:00+00:00https://blog.aloni.org/posts/bash-path-wrapping-executables/<p>When fixing up complex systems to our advantage we sometimes need to hook on
the intermediate execution of a program. The UNIX <code>PATH</code> environment is the
venerable search path for programs available in the environment. In this post
I'll discuss how to use to wrap around the execution of a program.</p>
<span id="continue-reading"></span>
<p>Previous post: <a href="../bash-functional-command-relay">#2</a>.</p>
<p>The <code>PATH</code> environment variable is a <code>:</code>-delimited strings of paths that are
searched in order for executables. For example, a build system may search
for the <code>gcc</code> compiler in <code>PATH</code>.</p>
<p>Let's wrap around <code>gcc</code>. To create our <code>gcc</code> wrapper, we need to decided
on a directory on which to place it. The name of the wrapping executable will
also be <code>gcc</code> so that it gets picked by <code>PATH</code> lookup. The absolute path
of that directory needs to be prepended to <code>PATH</code> for the wrapper to be found.
There are various ways to do this. The most common are:</p>
<ul>
<li>Prepend in shell command execution <code>PATH="/directory/to-wrapper:$PATH" <command></code>, only affecting that command.</li>
<li>Modify for the current shell script or interactive shell using <code>export PATH="/diretory/to-wrapper:$PATH"</code></li>
</ul>
<p>Our wrapper can be written as such:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source shell bash"><span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell">!/bin/bash</span><span class="comment line number-sign shell">
</span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Remove ourselves from $PATH, to prevent infinite</span><span class="comment line number-sign shell">
</span><span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> recursion when the wrapped executable is executed</span><span class="comment line number-sign shell">
</span><span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> by us.</span><span class="comment line number-sign shell">
</span><span class="variable other readwrite assignment shell">curdir</span><span class="keyword operator assignment shell">=</span><span class="string unquoted shell"><span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="variable function shell">realpath</span></span><span class="meta function-call arguments shell"> <span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="variable function shell">dirname</span></span><span class="meta function-call arguments shell"> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section expansion parameter begin shell">{</span></span><span class="meta group expansion parameter shell"><span class="variable other readwrite shell">BASH_SOURCE</span></span><span class="meta group expansion parameter shell"><span class="punctuation section expansion parameter end shell">}</span></span></span><span class="punctuation section parens end shell">)</span></span></span><span class="punctuation section parens end shell">)</span></span></span>
<span class="variable other readwrite assignment shell">path_tmp</span><span class="keyword operator assignment shell">=</span><span class="string unquoted shell"><span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable other readwrite shell">PATH</span></span></span> <span class="keyword operator logical pipe shell">|</span> <span class="meta function-call shell"><span class="variable function shell">tr</span></span><span class="meta function-call arguments shell"> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>:<span class="punctuation definition string end shell">'</span></span> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>\n<span class="punctuation definition string end shell">'</span></span></span> <span class="keyword operator logical pipe shell">|</span> <span class="punctuation separator continuation line shell">\
</span> <span class="meta function-call shell"><span class="variable function shell">awk</span></span><span class="meta function-call arguments shell"><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>v</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>cwd=<span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section expansion parameter begin shell">{</span></span><span class="meta group expansion parameter shell"><span class="variable other readwrite shell">curdir</span></span><span class="meta group expansion parameter shell"><span class="punctuation section expansion parameter end shell">}</span></span><span class="punctuation definition string end shell">"</span></span> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>$0!=cwd<span class="punctuation definition string end shell">'</span></span></span> <span class="keyword operator logical pipe shell">|</span> <span class="punctuation separator continuation line shell">\
</span> <span class="meta function-call shell"><span class="variable function shell">tr</span></span><span class="meta function-call arguments shell"> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>\n<span class="punctuation definition string end shell">'</span></span> <span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span>:<span class="punctuation definition string end shell">'</span></span></span><span class="punctuation section parens end shell">)</span></span></span>
<span class="meta function-call shell"><span class="storage modifier shell">export</span> <span class="variable other readwrite assignment shell">PATH</span><span class="keyword operator assignment shell">=</span><span class="string unquoted shell"><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section expansion parameter begin shell">{</span></span><span class="meta group expansion parameter shell"><span class="variable other readwrite shell">path_tmp<span class="keyword operator expansion shell">%</span></span></span><span class="meta group expansion parameter shell">:</span><span class="meta group expansion parameter shell"><span class="punctuation section expansion parameter end shell">}</span></span></span></span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Don't let Ctrl-C kill the script</span><span class="comment line number-sign shell">
</span><span class="meta function shell"><span class="storage type function shell">function</span> <span class="entity name function shell">ctrl_c</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span> <span class="meta function-call shell"><span class="support function colon shell">:</span></span><span class="keyword operator logical continue shell">;</span> <span class="punctuation section braces end shell">}</span></span>
<span class="meta function-call shell"><span class="support function trap shell">trap</span></span><span class="meta function-call arguments shell"> ctrl_c INT</span>
<span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>Before execution<span class="punctuation definition string end shell">"</span></span></span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Execute the original program</span><span class="comment line number-sign shell">
</span><span class="meta function-call shell"><span class="variable function shell"><span class="meta group expansion command parens shell"><span class="punctuation definition variable shell">$</span><span class="punctuation section parens begin shell">(</span><span class="meta function-call shell"><span class="variable function shell">basename</span></span><span class="meta function-call arguments shell"> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable other readwrite shell">0</span></span></span><span class="punctuation section parens end shell">)</span></span></span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">@</span></span><span class="punctuation definition string end shell">"</span></span></span>
<span class="variable other readwrite assignment shell">e</span><span class="keyword operator assignment shell">=</span><span class="string unquoted shell"><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">?</span></span></span> <span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Save exit status</span><span class="comment line number-sign shell">
</span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Clear out handler</span><span class="comment line number-sign shell">
</span><span class="meta function-call shell"><span class="support function trap shell">trap</span></span><span class="meta function-call arguments shell"> - INT</span>
<span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>After execution<span class="punctuation definition string end shell">"</span></span></span>
<span class="meta function-call shell"><span class="support function exit shell">exit</span></span><span class="meta function-call arguments shell"> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable other readwrite shell">e</span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>Notes on what is being done above:</p>
<ul>
<li>Important to clear out the directory in which the wrapper resides from <code>PATH</code>, otherwise we can cause an infinite recursion.</li>
<li>Forward the exit status of the original program.</li>
<li>Allow logic to be implemented before and following execution.</li>
<li>We can control how the original program is executed.</li>
<li>We can wrap more than one executable with a single wrapper source.</li>
</ul>
<h3 id="demo">Demo</h3>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ ls -l bin
total 4
lrwxrwxrwx 1 dan dan 7 Feb 19 09:18 gcc -> wrapper
lrwxrwxrwx 1 dan dan 7 Feb 19 09:18 ls -> wrapper
-rwxrwxr-x 1 dan dan 389 Feb 19 09:15 wrapper
$ export PATH=$(pwd)/bin:$PATH
$ ls bin
Before execution
gcc ls wrapper
After execution
$ gcc non-existant.c
Before execution
gcc: error: non-existant.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
After execution
</span></td></pre></tr></tbody></table></div><h2 id="final-note">Final note</h2>
<p>We should be careful about emitting to stdout and stderr by the wrapping code,
and may want to even avoid it completely, lest we break assumptions being made
by the upper-level scripts that execute the program we are wrapping.</p>
Rust Turbofish: Closure Return Type2022-02-05T00:00:00+00:002022-02-05T00:00:00+00:00https://blog.aloni.org/posts/rust-turbofish-closure-return-type/<p>There is a compiler type error in Rust that can happens when you try to pass a closure
that returns a <code>Result</code> type.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">error[E0282]: type annotations needed
--> src/main.rs:27:9
|
27 | Ok(())
| ^^ cannot infer type for type parameter `E` declared on the enum `Result`
</span></td></pre></tr></tbody></table></div>
<p>The clue to how to solve this is already in the error message - we need to add type
annotations. But why does this happen?</p>
<p>Suppose you have a wrapper that accepts a closure to be wrapped in execution:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">your_wrapper</span></span><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span>F, R<span class="punctuation definition generic end rust">></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="variable parameter rust">f</span><span class="punctuation separator rust">:</span> F</span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> R
</span></span><span class="meta function rust"><span class="keyword other rust">where</span>
F<span class="punctuation separator rust">:</span> FnOnce<span class="punctuation section group begin rust">(</span><span class="punctuation section group end rust">)</span> -> R,
</span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Before<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="storage type rust">let</span> v <span class="keyword operator assignment rust">=</span> <span class="support function rust">f</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>After<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
v
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>When using this wrapper, we can have two error types and use the wonderful
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://crates.io/crates/thiserror"><code>thiserror</code><span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> crate to implement a
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://doc.rust-lang.org/std/convert/trait.From.html"><code>From</code><span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> trait between
them.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="keyword other rust">use</span> <span class="meta path rust">thiserror<span class="punctuation accessor rust">::</span></span>Error<span class="punctuation terminator rust">;</span>
<span class="meta annotation rust"><span class="punctuation definition annotation rust">#</span><span class="punctuation section group begin rust">[</span><span class="variable annotation rust">derive</span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span></span><span class="meta annotation parameters rust"><span class="meta group rust">Error<span class="punctuation separator rust">,</span> Debug</span></span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="punctuation section group end rust">]</span></span>
<span class="meta enum rust"><span class="storage type enum rust">enum</span> <span class="entity name enum rust">Error</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span></span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta annotation rust"><span class="punctuation definition annotation rust">#</span><span class="punctuation section group begin rust">[</span><span class="variable annotation rust">derive</span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span></span><span class="meta annotation parameters rust"><span class="meta group rust">Error<span class="punctuation separator rust">,</span> Debug</span></span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="punctuation section group end rust">]</span></span>
<span class="meta enum rust"><span class="storage type enum rust">enum</span> <span class="entity name enum rust">MainError</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="meta annotation rust"><span class="punctuation definition annotation rust">#</span><span class="punctuation section group begin rust">[</span><span class="variable annotation rust">error</span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span></span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Sub error: {0}<span class="punctuation definition string end rust">"</span></span></span></span><span class="meta annotation parameters rust"><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="punctuation section group end rust">]</span></span>
Sub<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta annotation rust"><span class="punctuation definition annotation rust">#</span><span class="punctuation section group begin rust">[</span><span class="variable annotation rust">from</span><span class="punctuation section group end rust">]</span></span> Error</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation separator rust">,</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">main</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> <span class="meta generic rust"><span class="support type rust">Result</span><span class="punctuation definition generic begin rust"><</span><span class="punctuation section group begin rust">(</span><span class="punctuation section group end rust">)</span>, MainError<span class="punctuation definition generic end rust">></span></span></span> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type rust">let</span> r <span class="keyword operator assignment rust">=</span> <span class="support function rust">your_wrapper</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta function closure rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">|</span></span></span><span class="meta function closure rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">|</span></span> </span><span class="meta function closure rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Inside wrapper<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="support type rust">Ok</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="keyword operator rust">?</span><span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Following call<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="support type rust">Ok</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span>r</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>The limitation stems from how the <code>main</code> function is written. The <code>Result</code> is
deconstructed and reconstructed back. However, there is no way for the compiler
to come up a concrete type for the <code>Ok(())</code> value, and this is because no error
value or type signature was involved in the definition of the closure.</p>
<h2 id="turbofish-syntax-to-the-rescue">Turbofish syntax to the rescue</h2>
<p>This is where the turbofish syntax comes handy. The fix is to define the
closure as such:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="storage type rust">let</span> r <span class="keyword operator assignment rust">=</span> <span class="support function rust">your_wrapper</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta function closure rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">|</span></span></span><span class="meta function closure rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">|</span></span> </span><span class="meta function closure rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Inside wrapper<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="support type rust">Ok</span><span class="punctuation separator rust">:</span><span class="punctuation separator rust">:</span><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span><span class="keyword operator rust">_</span>, Error<span class="punctuation definition generic end rust">></span></span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="keyword operator rust">?</span><span class="punctuation terminator rust">;</span>
</span></td></pre></tr></tbody></table></div>
<p>We can use this to specify the type of the error without repeating the type of
the value or the identifier <code>Result</code>. This pattern is often repeated, so I have
casually defined an <code>Ok::<_, Error>(())</code> snippet in my editor for quick typing.</p>
Two Endgames For Blockchain and Cryptocurrencies2021-12-07T00:00:00+00:002021-12-07T00:00:00+00:00https://blog.aloni.org/posts/two-endgames-for-blockchain-and-cryptocurrencies/<p><b>tl;dr: HackerNews is boiling daily with debates over the merit (or
lack-thereof) of cryptocurrencies, bitcoin, and NFTs. I think there are two
opposite endgames - all or nothing.</b></p>
<p>The determining factor for widespread maturity of a technology is whether it is
necessary for modern quality of life.</p>
<p>Electricity, running water, internet in general, internet-based delivery of
transportation (Uber), internet-based delivery of goods (Amazon), cars. These
are technologies that work very well for people. A medium to exchange good for
the represented value of these products and technologies: state-issued
currency, credit card, bank account. These also work very well.</p>
<p>So as for blockchain (the tech) and cryptocurrencies (the value exchange), I
see two possible futures:</p>
<ol>
<li>
<p>Cryptocurrency and/or blockchain-based technologies (stake/work) wind up
as a part of the daily life of almost every person in the first world's
day-to-day life, just like the technologies I listed earlier. Whether I like
it or not, I <em>have</em> to use it.</p>
</li>
<li>
<p>Realizing that Scenario [1] would never come, the jig is up. At some point
enthusiasts realize this and their online community shrink back to year 2015 or
earlier levels ; cryptocurrencies value in state-issued fiat currencies also set
back to the same levels.</p>
</li>
</ol>
<p>Is Scenario [1] getting closer any moment? Depends on whom you ask. Except for
enthusiasm among my peers, I don't see any actual evidence for this yet, and
despite the consistent 10-year increase of value of cryptocurrencies, we may be
heading into Scenario [2] still.</p>
<p>I don't invest in cryptocurrencies and have no use for them. I don't use any
blockchain-based technology. It would remain that way indefinitely for me, if
and only if scenario [2] is realized.</p>
<p>We can also measure the value of a technology using the <i>shutdown test</i>.
If in 2021 we shut down electricity for everyone - the result is terrible. If
we shut down all banks, we get societal turmoil. But if hypothetically we shut
down Bitcoin and Ethereum, by average it would only be equivalent to a small
stock market downturn to some people, and a loss of investment to various
speculators.</p>
<hr />
<p>One quality about the opinion I bring here is that it is regardless of whether
I could have invested in cryptocurrencies and made millions. I think
there is no prejudice, and it would not have affected it.</p>
Bash #2 - Functional Command Relay2021-04-26T00:00:00+00:002021-04-26T00:00:00+00:00https://blog.aloni.org/posts/bash-functional-command-relay/<p>In this post I introduce a nice use for functions in <code>bash</code>.</p>
<span id="continue-reading"></span>
<p>Previous post: <a href="../bash-execution-relay">#1</a>.</p>
<p>In software projects, would often appear a 'scripts' directory with various
scripting utilities. The inexperienced <code>bash</code> scriptwriter would usually not
use functions at all and instead litter up the directory with many execution
entry points such as:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">build.sh
run.sh
deploy.sh
</span></td></pre></tr></tbody></table></div>
<p>If we would like to clean this up and combines these functions to a single
file, say <code>command.sh</code>, how one would do so?</p>
<p>A nice way to accomplish is using the following:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source shell bash"><span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell">!/bin/bash</span><span class="comment line number-sign shell">
</span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> ... common code here ...</span><span class="comment line number-sign shell">
</span>
<span class="meta function shell"><span class="entity name function shell">build</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Original content and logic of bash.sh</span><span class="comment line number-sign shell">
</span><span class="punctuation section braces end shell">}</span></span>
<span class="meta function shell"><span class="entity name function shell">run</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Original content and logic of run.sh</span><span class="comment line number-sign shell">
</span><span class="punctuation section braces end shell">}</span></span>
<span class="meta function shell"><span class="entity name function shell">deploy</span><span class="punctuation section parens begin shell">(</span><span class="punctuation section parens end shell">)</span> <span class="punctuation section braces begin shell">{</span>
<span class="comment line number-sign shell"><span class="punctuation definition comment begin shell">#</span></span><span class="comment line number-sign shell"> Original content and logic of deploy.sh</span><span class="comment line number-sign shell">
</span><span class="punctuation section braces end shell">}</span></span>
<span class="meta function-call shell"><span class="variable function shell"><span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">@</span></span><span class="punctuation definition string end shell">"</span></span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>(for clarification regarding <code>"$@"</code>, see the previous post regarding <a href="/posts/bash-execution-relay/">execution relay</a>)</p>
<p>Here is our example execution:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ ./command.sh build <params>
<output of the original build logic>
</span></td></pre></tr></tbody></table></div>
<p>Functions in <code>bash</code> behave like mini-scripts in the same manner that original
scripts would behave, and similar to 'real' executable programs that are
already reachable from <code>$PATH</code>. It is very convenient that <code>bash</code> functions are
inter-changeable with regular programs. For instance, they can be used in shell
pipe components and in sub-shells and under <code>$(..)</code>.</p>
<p>I call the method above the Functional Command Relay pattern, as it allows
execution of each function with the parameters to these functions relayed
as-is. It can even be used recursively to represent a command tree in script
command line parsing.</p>
<p>A natural extension to this method is choosing how strict we are with the
command itself, which is the first parameter to the main script. For example,
we can replace the last <code>"$@"</code> line with the following:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source shell bash"><span class="meta conditional case shell"><span class="keyword control conditional case shell">case</span> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable other readwrite shell">1</span></span> <span class="keyword control in shell">in</span>
<span class="meta conditional case clause patterns shell">build<span class="keyword operator logical shell">|</span>run<span class="keyword operator logical shell">|</span>deploy<span class="keyword control conditional patterns end shell">)</span></span><span class="meta conditional case clause commands shell"> <span class="meta function-call shell"><span class="variable function shell"><span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">@</span></span><span class="punctuation definition string end shell">"</span></span></span></span> </span><span class="meta conditional case clause commands shell"><span class="punctuation terminator case clause shell">;;</span></span>
<span class="meta conditional case clause patterns shell"><span class="string quoted single shell"><span class="punctuation definition string begin shell">'</span><span class="punctuation definition string end shell">'</span></span><span class="keyword control conditional patterns end shell">)</span></span><span class="meta conditional case clause commands shell"> <span class="meta function-call shell"><span class="variable function shell">print_help</span></span><span class="keyword operator logical continue shell">;</span> <span class="meta function-call shell"><span class="support function exit shell">exit</span></span><span class="meta function-call arguments shell"> -1</span> </span><span class="meta conditional case clause commands shell"><span class="punctuation terminator case clause shell">;;</span></span>
<span class="meta conditional case clause patterns shell"><span class="keyword operator regexp quantifier shell">*</span><span class="keyword control conditional patterns end shell">)</span></span><span class="meta conditional case clause commands shell"> <span class="meta function-call shell"><span class="support function echo shell">echo</span></span><span class="meta function-call arguments shell"> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>Invalid command '<span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable other readwrite shell">1</span></span>'<span class="punctuation definition string end shell">"</span></span></span> <span class="keyword operator logical continue shell">;</span> <span class="meta function-call shell"><span class="support function exit shell">exit</span></span><span class="meta function-call arguments shell"> -1</span> </span><span class="meta conditional case clause commands shell"><span class="punctuation terminator case clause shell">;;</span></span>
<span class="keyword control conditional end shell">esac</span></span>
</span></td></pre></tr></tbody></table></div>Bash #1 - Execution Relay2021-04-25T00:00:00+00:002021-04-25T00:00:00+00:00https://blog.aloni.org/posts/bash-execution-relay/<p>The <code>bash</code> shell is somewhat like the lingua-franca of the UNIX-based shell
scripting world, as nothing else manages to displace it. The usefulness of a
language is significant if it is omnipresent.</p>
<p>In this post I begin a series about programming in <code>bash</code>. Its earlier
incarnation <code>sh</code> is less of a concern to me because <code>bash</code> can be found in
almost in every place where <code>sh</code> is present.</p>
<span id="continue-reading"></span>
<p>The pitfalls in <code>bash</code>-programming for the initiate or non-frequent shell
script programmer are numerous. However, they are easy to avoid by just
sticking in to some rules.</p>
<h2 id="relaying-execution">Relaying execution</h2>
<p>Occasionally you want to relay an execution of one program to another. Possible reasons: providing arguments, environment variable hacks, path, and so
forth.</p>
<p>For example, let's create a mutex around another program using <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://linux.die.net/man/1/flock"><code>flock</code><span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>,
a program to manage locks from shell scripts.</p>
<p><code>flock</code> has the following syntax:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">flock [options] file|directory command [arguments]
</span></td></pre></tr></tbody></table></div>
<p>From this, it is apparent that we would need to relay arguments to the executed program.</p>
<h2 id="relaying-arguments-always-use">Relaying arguments (always use <code>"$@"</code>!)</h2>
<p>It is important to understand the difference between the following expressions:</p>
<ul>
<li><code>$*</code></li>
<li><code>$@</code></li>
<li><code>"$*"</code></li>
<li><code>"$@"</code></li>
</ul>
<p>Let's test this with a simple <code>bash</code> script. The script executes Python to see
what list of strings was actually relayed as an argument list.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source shell bash"><span class="meta function-call shell"><span class="support function echo shell">echo</span></span>
<span class="meta function-call shell"><span class="variable function shell">python</span></span><span class="meta function-call arguments shell"><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>c</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>import sys; print(sys.argv[1:])<span class="punctuation definition string end shell">"</span></span> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">*</span></span></span>
<span class="meta function-call shell"><span class="variable function shell">python</span></span><span class="meta function-call arguments shell"><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>c</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>import sys; print(sys.argv[1:])<span class="punctuation definition string end shell">"</span></span> <span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">@</span></span></span>
<span class="meta function-call shell"><span class="variable function shell">python</span></span><span class="meta function-call arguments shell"><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>c</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>import sys; print(sys.argv[1:])<span class="punctuation definition string end shell">"</span></span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">*</span></span><span class="punctuation definition string end shell">"</span></span></span>
<span class="meta function-call shell"><span class="variable function shell">python</span></span><span class="meta function-call arguments shell"><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>c</span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span>import sys; print(sys.argv[1:])<span class="punctuation definition string end shell">"</span></span> <span class="string quoted double shell"><span class="punctuation definition string begin shell">"</span><span class="meta group expansion parameter shell"><span class="punctuation definition variable shell">$</span><span class="variable language shell">@</span></span><span class="punctuation definition string end shell">"</span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>Execution result:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ bash test.sh foo bar 'hello world' \?
['foo', 'bar', 'hello', 'world', '1', '2', 'm']
['foo', 'bar', 'hello', 'world', '1', '2', 'm']
['foo bar hello world ?']
['foo', 'bar', 'hello world', '?']
</span></td></pre></tr></tbody></table></div>
<p>From this we can see there's only one good option. Let's break it down to what happens:</p>
<ul>
<li><code>$*</code> - don't quote arguments, perform wildcard expansion and then relay</li>
<li><code>$@</code> - same</li>
<li><code>"$*"</code> - relay all arguments as one argument with <code>IFS</code> separator concatenation (space if undefined), don't perform wildcard expansion</li>
<li><code>"$@"</code> - relay arguments and quote them, don't expand, equivalent to <code>"$1" "$2" "$3" ...</code></li>
</ul>
<p>There is more information about this in <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Special-Parameters">bash's docs<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<h2 id="avoiding-a-redundant-shell-process">Avoiding a redundant shell process</h2>
<p>Once we reach executing the wrapper program, the shell process has done its
job. Therefore, it would be best to replace it with the wrapped program. This
can be done using an <code>exec</code> prefix.</p>
<h2 id="full-example">Full example</h2>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">#!/bin/bash
mkdir -p ~/.locker || exit -1
exec flock ~/.locker/my-lock-file "$@"
</span></td></pre></tr></tbody></table></div>Gitology #2 - git-retext2020-10-23T00:00:00+00:002020-10-23T00:00:00+00:00https://blog.aloni.org/posts/gitology-2-git-retext/<p>This is the second post in a series to expand on various utilities I wrote to
assist my work with Git. Some of these utilities are located in a repository
on GitHub called <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/da-x/misc-gitology">misc-gitology<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<p>Previous post: <a href="../gitology-1-git-flip-history">#1</a>.</p>
<p>Today I'll introduce the commit rewriter - <code>git-retext</code>.</p>
<span id="continue-reading"></span><h2 id="familiar-ways-of-git-history-editing">Familiar ways of Git history editing</h2>
<p>In the previous post I mentioned the desire to present a clear Git history to
reviewers. Sometimes, we are reviewing our own unpublished Git history and
would like to do some polishing here and there before pushing it to a
remote repository.</p>
<p>There are multitude of ways in which editing the history can be done:</p>
<ul>
<li><code>git commit --amend</code>, with or without <code>-a</code> (in the later <code>git add</code> is used). This
takes care of the <code>HEAD</code> commit, but not commits that are earlier to it.</li>
<li><code>git rebase</code>, where you labor on a fixup commit or a set of them,
with or without <code>--interactive</code>, with or without <code>--autosquash</code>, so that
the fixup changes amend commits further down the history. There are tools such
as <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/tummychow/git-absorb">git-absorb<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> that can be used to
automate creation of fixup commits.</li>
<li><code>git-filter-branch</code>, a big hammer that lets you run a command per a commit,
where the result of a command is the rewritten tree. Obviously not
quite suitable for easy 'final touches'.</li>
<li>Editor environment features. In Emacs, <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://magit.vc/">Magit<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, and in
Vim, <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/jreybert/vimagit">Vimagit<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, and possibly other editors
have their own Git integrations, some are elaborate enough to ease on editing
the history. However, to each his own, and belaboring on a certain editor
environment would get us off-topic.</li>
</ul>
<h2 id="a-new-way-git-retext">A new way - <code>git-retext</code></h2>
<p>In the process of reviewing, we are most likely looking at diffs. What if it
was possible in an environment-independent way, to just edit the diff? </p>
<p>In the pre-cursor to the invention of Git, developers used emails (and some are
still using emails) in order to pass along changes to one another. The <code>git format-patch</code> command knows how to turn a list of commits into such emails. As
plain text emails can be easily edited by a text editor, it stands to reason
that we can edit changes in-place, if we momentarily turn commits into emails
and back.</p>
<p>This is what <code>git-retext</code> essentially lets you do.</p>
<p>For example, we want to edit the most recent commit in the history. Let's issue
the command:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">git-retext HEAD~1
</span></td></pre></tr></tbody></table></div>
<p>This <code>git-retext</code> command turns the <code>HEAD~1..HEAD</code> range into emails, and then
we immediately find ourselves thrown into editing in the default text editor
that we configured either in Git's configuration, or <code>EDITOR</code> environment
variable. The "thrown into editor" situation should not be foreign to Git
users, as it is done in the default <code>git commit</code> or <code>git rebase -i</code> workflows.</p>
<p>Here's an example for such <code>git-retext HEAD~1</code> invocation at one of my
repositories:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">From bbb92d7cc4ac33bd0e368164d555e4e93a66e658 Mon Sep 17 00:00:00 2001
From: Dan Aloni <alonid@gmail.com>
Date: Thu, 21 May 2020 09:56:01 +0300
Subject: [PATCH] Apply random rotation for new piece
By applying a random set of rotation, the new piece is effectively
rotated to all possible rotations.
---
src/main.rs | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/main.rs b/src/main.rs
index de5aba608ec3..d2741fa61fe2 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -161,6 +161,9 @@ impl Game {
self.falling = self.possible_pieces[idx].clone();
self.shift = (0, 0);
+ for _ in 0 .. rng.gen_range(0, 4usize) {
+ self.rotate(false)
+ }
}
fn render(&self, window: &mut PistonWindow, event: &Event) {
--
2.26.2
</span></td></pre></tr></tbody></table></div>
<p>This is how a commit looks like when it is sent by email. It is possible to
edit the first line of the commit in the <code>Subject: </code> ; and the details of the
commit message before the <code>---</code>, and the diff itself. About the diff, we don't
have to fix the unidiff's meta-data, because <code>git-retext</code>'s processing will
take care of it instead.</p>
<p>Saving the temporary and exiting, <code>git-retext</code> will try to apply the edited
commits using <code>git am</code>. If we have carefully edited them, they should
apply cleanly. The resultant <code>HEAD</code> is the edited history.</p>
<p>Of course, this new method is not perfect, but it's a time saver for certain
set of editing activities. After getting accustomed to it, you may start
reviewing changes in <code>git-retext</code> even if you are not ending up modifying them,
instead of running <code>git-diff</code> or viewing a diff made by the editor environment.</p>
<h2 id="advantages">Advantages</h2>
<ul>
<li>Easy to amend changes right when they are being reviewed, as long as
the editing is not too complicated. Adding new diff lines inside hunks
is possible.</li>
<li>Easy to do search and replace on the changes themselves, and that also takes
care of the commit message, and added filenames if relevant.</li>
<li>It is possible to remove unwanted diff hunks.</li>
<li>Splitting of commits to small new commits is possible by adding a new email
header in between hunks (no need to edit the diffstat).</li>
</ul>
<h2 id="disadvantages">Disadvantages</h2>
<ul>
<li>If there is more than one commit, and the commits are dependent on one
another, we must be more careful so that the editing is consistent, and that
the diff hunk context is correct.</li>
<li>It's easy to mess up the editing and discover the mess only when <code>git-retext</code>
fails to apply the changes.</li>
</ul>
<p>Future enhancements of <code>git-retext</code> should assist in avoiding mistakes during
editing, perhaps with better editor integration.</p>
<h1 id="how-it-is-implemented">How it is implemented</h1>
<p><code>git-retext</code> is Python 3 script that relies on <code>recountdiff</code> in order to fix
the edited diffs. The underlying git commands being used are
<code>git-format-patch</code>, <code>git-am</code>, and <code>git-reset</code>.</p>
Gitology #1 - git-flip-history2020-10-16T00:00:00+00:002020-10-16T00:00:00+00:00https://blog.aloni.org/posts/gitology-1-git-flip-history/<p>This is the first post in a series to expand on various utilities I wrote to
assist my work with Git. Some of these utilities are located in a repository
on GitHub called <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/da-x/misc-gitology">misc-gitology<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<p>Today I'll introduce the history flipper - <code>git-flip-history</code>.</p>
<span id="continue-reading"></span><h2 id="the-problem-with-splitting-commits">The problem with splitting commits</h2>
<p>When working with Git and browsing other developers' commit history, it is
clear to many developers that separating logical changes to commits often
provides value to reviewers. It also makes it easier to revert changes if
needed, or when bisecting for bugs.</p>
<p>However, it is often hard to abide by the rule. When we are about to make a
change, sometimes we find ourselves refactoring or doing more changes along the
way, and we end up doing many preceding and/or proceeding changes to support
the change we wanted to do in the first place.</p>
<p>Managing several closely related changes together while they are still "hot in
the stove" really depends on development style and organization of the person
doing so. Some manage to take every change to its own branch and commit it
right away, nice and cleanly. Others have the discipline to split right away to
separate commits and rebase-squash them with fixups. However, others cannot
afford many of the context switches that would be involved and prefer to delay
"feature splitting" to the very end once everything feels more mature and
well-formed. The utility discussed in this post is more helpful to the latter
group of developers.</p>
<p>Splitting a large commit that is currently in <code>HEAD</code> involves doing some Git
maneuvering starting with <code>git reset</code>, proceeding to <code>git add</code> and <code>git add -p</code>, and an occasional <code>git commit</code>. However, this has a limitation - some
logical changes may be dependent on one another, in this case <code>git add -p</code>
would not help. For example, suppose two features add to a global list, where
they modify the same location in a file.</p>
<h2 id="history-flipping-to-the-rescue">History flipping to the rescue</h2>
<p>Another way to tackle the problem of untangling features from one another is
this - if I can do the work to remove the features one by one, it would be the
mirror image of the work of rewriting these features from scratch, and in that
case I just need an elaborate Git trick to flip the history on itself!</p>
<p>In further detail, I have a large commit implementing A+B+C. If I write a new
commit that reverts A, and another one that reverts B, and flip the history of
the three versions, the result would be a history that implements all three
versions separately.</p>
<p>So this is where <code>git-flip-history</code> comes into play, where it also makes sure
that the commit log looks sane afterward.</p>
<p>For example, we have the following commit, with a tentative commit message:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">219bd740a8f1 WIP - three changes packed into one
</span></td></pre></tr></tbody></table></div>
<p>Our purpose is to reach a state where we have three commits each describing
a separate feature (can also be a bugfix any other kind of change).</p>
<p>Here, we'll manually revert the first two features into two commits, and for
each one put specially crafted commit message with a <code>Revert: </code> prefix. This
can be done in the editor, and <code>git checkout</code> can also assist. Once we're done
for the original commit we'll reword it, so it becomes prefixed with a <code>First: </code>
prefix, describing the remaining feature which we have <strong>not</strong> reverted. (i.e.
whatever change the diff to <code>HEAD~3</code> presented). We are free to write full
commit messages, as long as the <code>Revert: </code> and <code>First: </code> prefixes are there.</p>
<p>Following our work to revert the features and reword the commits accordingly,
our history looks like this:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">509c3964befd Revert: Ignore type aliases
8941699bfb69 Revert: Treat DefKind::Mod as being in the value namespace
372d10a7ffcc First: Have trait names on their own namespace
</span></td></pre></tr></tbody></table></div>
<p>Now that we are ready, we run <code>git-flip-history</code>. This requires no user input,
and now <code>HEAD</code> looks like this:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">a086ec0322a4 Treat DefKind::Mod as being in the value namespace
9d78613ed775 Ignore type aliases
86d20b3e77dc Have trait names on their own namespace
</span></td></pre></tr></tbody></table></div>
<p>Observing the changes with <code>git log -p</code>, we should see a sane history now,
<strong>where each commit adds the feature that the commit message talks about</strong>.
It's worth to stress out that flipping the history like this does not require
us to solve any conflicts, unlike with <code>git rebase -i</code> in the case where
changes overlap on file offsets. Untangling the changes from one another is
done only in the process of creating the <code>Revert: </code> commits, and you are free
to use whatever method e.g. <code>git checkout -p</code>, or manual editing, in order to
create these commits.</p>
<h3 id="further-discussion">Further discussion</h3>
<p>To explain how this is possible, let's look at it from another angle.</p>
<p>We have the following history:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">HEAD Revert A This tree has C
HEAD~1 Revert B This tree has A+C
HEAD~2 Implement A+B+C This tree has A+B+C
</span></td></pre></tr></tbody></table></div>
<p>By flipping, we now have this (start by comparing the third column):</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">HEAD Implement B This tree has A+B+C
HEAD~1 Implement A This tree has A+C
HEAD~2 Implement C This tree has C
</span></td></pre></tr></tbody></table></div><h2 id="how-does-it-work">How does it work</h2>
<p>The <code>git-flip-history</code> program is a bash script that looks back at the history,
finding the <code>Revert: </code> prefixes and stops when seeing a commit with the <code>First: </code> prefix. The history recreation process does not do any working tree changes,
by simply invoking the <code>git-commit-tree</code> command. The resulting topmost version
is identical to the original branch version, and the branch is reset using
<code>git reset --hard</code> to the new history. The script currently requires a clean
Git status for good measure (this may change in the future).</p>
Path Trimming In Nightly Rust2020-09-04T00:00:00+00:002020-09-04T00:00:00+00:00https://blog.aloni.org/posts/path-trimming-in-rust-nightly/<p>As of yesterday, the <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/rust-lang/rust/pull/73996">Rust PR<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>
(which I had worked on) is merged into Rust nightly, and it has wide
implications on compiler errors.</p>
<p>In this post I describe the change and what to expect from it.</p>
<span id="continue-reading"></span><h2 id="the-problem-with-full-paths-in-errors">The problem with full paths in errors</h2>
<p>A simple program such as the following, would result in a type error.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">main</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type rust">let</span> a <span class="keyword operator assignment rust">=</span> <span class="support macro rust">vec!</span><span class="meta group rust"><span class="punctuation section group begin rust">[</span><span class="support macro rust">vec!</span><span class="meta group rust"><span class="punctuation section group begin rust">[</span><span class="support type rust">String</span><span class="meta path rust"><span class="punctuation accessor rust">::</span></span>from<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>a<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation section group end rust">]</span></span><span class="punctuation section group end rust">]</span></span><span class="punctuation terminator rust">;</span>
<span class="storage type rust">let</span> b <span class="keyword operator assignment rust">=</span> <span class="support macro rust">vec!</span><span class="meta group rust"><span class="punctuation section group begin rust">[</span><span class="support type rust">String</span><span class="meta path rust"><span class="punctuation accessor rust">::</span></span>from<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>b<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation section group end rust">]</span></span><span class="punctuation terminator rust">;</span>
a <span class="keyword operator comparison rust">==</span> b<span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>The type error can be described as such: cannot compare between values of the
types <code>Vec<Vec<String>></code> and <code>Vec<String></code>. Before the changes in the PR, this
was <em>almost</em> the first line of the error message, and the rest of the error
message gives us more details about traits:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">old output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">error[E0277]: can't compare `std::vec::Vec<std::string::String>` with `std::string::String`
--> example.rs:5:7
|
5 | a == b;
| ^^ no implementation for `std::vec::Vec<std::string::String> == std::string::String`
|
= help: the trait `std::cmp::PartialEq<std::string::String>` is not implemented for `std::vec::Vec<std::string::String>`
= note: required because of the requirements on the impl of `std::cmp::PartialEq<std::vec::Vec<std::string::String>>` for `std::vec::Vec<std::vec::Vec<std::string::String>>`
</span></td></pre></tr></tbody></table></div>
<p>It is surely noticeable that in the above error the greatest contribution to
cognitive burden is the fully qualified paths (e.g. <code>std::vec::Vec</code>) of types
and traits. It has made a significant readability difference for many people.</p>
<h2 id="enter-path-trimming">Enter path trimming</h2>
<p>In the large majority of cases there would be only one <code>Vec</code> symbol and one
<code>String</code> symbol that is importable through the entire program being linked, for
all crates that are available. Surely that there are crates existing that
define items named <code>Vec</code>, but they are rare, and the situation that the user
defines <code>Vec</code> is rare.</p>
<p>Considering the observation that the overlap between module namespaces is
rather minimal, we can do a uniqueness check that verifies that uniqueness
holds, i.e, that <code>Vec</code> and <code>String</code> are unique as items defined in the
compilation. Even if that's not the case, the compilation would still succeed
without any new warning. But for the unique symbols, we don't have to print the
entire path in warnings and errors, and we can thus trim it to the last
component - the symbol itself.</p>
<p>With trimming according to uniqueness, the following error is printed instead:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">new output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">error[E0277]: can't compare `Vec<String>` with `String`
--> example.rs:5:7
|
5 | a == b;
| ^^ no implementation for `Vec<String> == String`
|
= help: the trait `PartialEq<String>` is not implemented for `Vec<String>`
= note: required because of the requirements on the impl of `PartialEq<Vec<String>>` for `Vec<Vec<String>>`
</span></td></pre></tr></tbody></table></div>
<p>This behavior can be controlled using a new debug option <code>-Z trim-diagnostic-paths=false</code>, and it is enabled by default only for rustc
itself.</p>
<p>As for the toll it takes on the compiler, it is similar to the algorithm that
computes 'use suggestions' on errors caused by undefined identifiers. This
means iterating all importable symbols of the entire program or library being
linked. Since this may be heavy, we made sure it is only done in case there
are warnings or errors by the compiler. If that assertion is invalidated,
it's a bug, and you'd see a panic related to trimmed paths.</p>
<h3 id="trimming-considerations">Trimming considerations</h3>
<p>Trimming is done only relative to what the currently built crate does, so -</p>
<ul>
<li>
<p>All the local definitions in the built crate are considered, regardless of
whether they are exported from it or not. This is different from how external
crates are treated, where only the externally visible and importable
definitions are taken into account.</p>
</li>
<li>
<p>Trimming is considered between all crates including the one being built, so
if you define a <code>Vec</code> type anywhere in your crate, then <code>Vec</code> name will no
longer be considered unique because another <code>Vec</code> can be imported from
<code>std::vec</code> too, and thus the full paths of both types will be printed as
just as before.</p>
</li>
<li>
<p>Because several glob imports (i.e. <code>use foo::*;</code>) can happen in a single
place, it wouldn't be clear which items they bring if we trim the paths that
are related to these items. Thus, glob imports cancel out the uniqueness of the
symbols that they import.</p>
</li>
</ul>
<h2 id="what-s-next">What's next</h2>
<p>This change in behavior will probably go under some refinements and more
testing until it reaches stable Rust. There are expected follow-ups, for
instance, to allow some ambiguity, as not all items are treated equal. For
example between the <code>Result</code> type alias in <code>std::io</code>, and the <code>Result</code> type
itself from <code>std::result</code>.</p>
<h2 id="thanks">Thanks</h2>
<p>The change has been hard to maintain as a PR, as it affected more than 1000
unit tests. It has gone through several revisions until the implementation was
good.</p>
<p>However, despite being not a frequent Rust compiler contributor where most of
the code involved was new to me, it has been greatly instructive to rely on
long-term members of the Rust compiler team. There have been folks who were
crucial in reviewing, so I'd like to thank them — <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/petrochenkov">Vadim
Petrochenkov<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/eddyb">Eduard
Burtescu<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/estebank"> Esteban
Kuber<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, and also other contributors — <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/Aaron1011">Aaron
Hill<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, and <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/lzutao">luzato<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>
for their help.</p>
Tracking Linux Kernel Sources of Distributions2020-02-01T00:00:00+00:002020-02-01T00:00:00+00:00https://blog.aloni.org/posts/tracking-linux-kernel-sources-of-distributions/<p><strong>TL;DR</strong>: I provide easily browsable CentOS kernel sources <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/kernelim/linux">on
GitHub<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<span id="continue-reading"></span>
<p>When developing Linux-based back-end systems, on some situations it is often
quite viable to look at <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.kernel.org">Linux kernel<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> sources in order
to understand problems. This is true for my clients who design high
performance and critical systems, often with kernel code added.</p>
<p>However, the source browsing provided by
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://elixir.bootlin.com/linux/latest/source">Elixir<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, though an
indispensable and easily accessible resource, does not cover the heavily
patched kernels that enterprise Linux users sometimes use. The clients that
need to do kernel development usually don't take upstream kernels as a base for
a good reason - the enterprise-patched kernels are more field-driven and
heavily patched with fixes and backported features.</p>
<p>For example, the kernels being maintained by <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://redhat.com">Red Hat<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>,
contain a large number of
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Backporting">backports<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> and changes. Due to
internal decisions, the sources of these kernels only come as source packages,
as the GPL does not enforce any kind of source control usage on released code.</p>
<p>Because companies such as Red Hat are reluctant to look into issues in
customized and patched kernels, organization that need to use such kernels
sometimes choose to use the equivalent <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.centos.org">CentOS<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> kernels
with the downside of supporting these kernels themselves, outside of Red Hat,
where exact Git histories of such kernels is not available.</p>
<p>In an ideal world, it would have been possible to browse the source of all the
Linux kernels of distributions in one place. However, the Linux kernel is
maintained in distribution forks, and the maintainship is fragmented. So to
compensate, and in order to ease my work, a few years ago I have decided to
artificially create Git trees that track these kernels release by release.</p>
<p>This semi-automated work is <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/kernelim/linux">reachable in
GitHub<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>. In the future, I may add more
information to each commit, for example, source RPM change-log entries. For
now, it allows to perform diffs between various minor and major kernels
versions of the distribution kernel, compare the distribution with the upstream
kernel, and other actions.</p>
A stack-less Rust coroutine library under 100 LoC2020-01-25T00:00:00+00:002020-01-25T00:00:00+00:00https://blog.aloni.org/posts/a-stack-less-rust-coroutine-100-loc/<p>As of stable <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.rust-lang.org/">Rust<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> 1.39.0, it is possible to
implement a very basic and safe
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Coroutine">coroutine<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> library using Rust's
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.infoq.com/presentations/rust-2019/"><code>async</code>/<code>await</code><span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> support, and
in under 100 lines of code. The implementation depends solely on <code>std</code> and is
stack-less (meaning, not depending on a separate CPU architecture stack).</p>
<span id="continue-reading"></span>
<p>A very basic simple coroutine library contains only an event-less 'yield'
primitive, which stops execution of the current coroutine so that other
coroutines can run. This is the kind of library that I chose to demonstrate in
this post to provide the most concise example.</p>
<h2 id="yielder">Yielder</h2>
<p>To the coroutine we pass a <code>Fib</code> struct that only contains a simple binary
state. This <code>Fib</code> struct has a <code>waiter</code> method that creates a
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://doc.rust-lang.org/std/future/trait.Future.html"><code>Future</code><span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> that the
coroutine can use in order to be <code>await</code>ed upon.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="keyword other rust">use</span> <span class="meta path rust">std<span class="punctuation accessor rust">::</span></span><span class="meta path rust">future<span class="punctuation accessor rust">::</span></span>Future<span class="punctuation terminator rust">;</span>
<span class="keyword other rust">use</span> <span class="meta path rust">std<span class="punctuation accessor rust">::</span></span><span class="meta path rust">pin<span class="punctuation accessor rust">::</span></span>Pin<span class="punctuation terminator rust">;</span>
<span class="keyword other rust">use</span> <span class="meta path rust">std<span class="punctuation accessor rust">::</span></span><span class="meta path rust">task<span class="punctuation accessor rust">::</span></span><span class="meta block rust"><span class="punctuation section block begin rust">{</span>Poll<span class="punctuation separator rust">,</span> Context</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span><span class="punctuation terminator rust">;</span>
<span class="meta enum rust"><span class="storage type enum rust">enum</span> <span class="entity name enum rust">State</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
Halted<span class="punctuation separator rust">,</span>
Running<span class="punctuation separator rust">,</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta struct rust"><span class="storage type struct rust">struct</span> </span><span class="meta struct rust"><span class="entity name struct rust">Fib</span> </span><span class="meta struct rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="variable other member rust">state</span><span class="punctuation separator type rust">:</span> State,
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta impl rust"><span class="storage type impl rust">impl</span> </span><span class="meta impl rust"><span class="entity name impl rust">Fib</span> </span><span class="meta impl rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">waiter</span></span><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span><span class="storage modifier lifetime rust">'a</span><span class="punctuation definition generic end rust">></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">&</span><span class="storage modifier lifetime rust">'a</span> <span class="storage modifier rust">mut</span> <span class="variable parameter rust">self</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> <span class="meta generic rust">Waiter<span class="punctuation definition generic begin rust"><</span><span class="storage modifier lifetime rust">'a</span><span class="punctuation definition generic end rust">></span></span></span> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
Waiter <span class="meta block rust"><span class="punctuation section block begin rust">{</span> fib<span class="punctuation separator rust">:</span> <span class="variable language rust">self</span> </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta struct rust"><span class="storage type struct rust">struct</span> </span><span class="meta struct rust"><span class="meta generic rust"><span class="entity name struct rust">Waiter</span><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span><span class="storage modifier lifetime rust">'a</span><span class="punctuation definition generic end rust">></span></span></span></span><span class="meta struct rust"> </span><span class="meta struct rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="variable other member rust">fib</span><span class="punctuation separator type rust">:</span> <span class="keyword operator rust">&</span><span class="storage modifier lifetime rust">'a</span> <span class="storage modifier rust">mut</span> Fib,
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta impl rust"><span class="storage type impl rust">impl</span></span><span class="meta impl rust"><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span><span class="storage modifier lifetime rust">'a</span><span class="punctuation definition generic end rust">></span></span></span><span class="meta impl rust"> Future <span class="keyword other rust">for</span></span><span class="meta impl rust"> <span class="entity name impl rust">Waiter</span><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span><span class="storage modifier lifetime rust">'a</span><span class="punctuation definition generic end rust">></span></span> </span><span class="meta impl rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type type rust">type</span> <span class="entity name type rust">Output</span> <span class="keyword operator assignment rust">=</span> <span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">poll</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="storage modifier rust">mut</span> <span class="variable parameter rust">self</span>: <span class="meta generic rust">Pin<span class="punctuation definition generic begin rust"><</span><span class="keyword operator rust">&</span><span class="storage modifier rust">mut</span> <span class="storage type rust">Self</span><span class="punctuation definition generic end rust">></span></span>, <span class="variable parameter rust">_cx</span><span class="punctuation separator rust">:</span> <span class="keyword operator rust">&</span><span class="storage modifier rust">mut</span> Context</span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> <span class="meta generic rust">Poll<span class="punctuation definition generic begin rust"><</span><span class="meta path rust"><span class="storage type rust"><span class="storage type rust">Self</span><span class="punctuation accessor rust">::</span></span></span>Output<span class="punctuation definition generic end rust">></span></span></span> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="keyword control rust">match</span> <span class="variable language rust">self</span><span class="punctuation accessor dot rust">.</span>fib<span class="punctuation accessor dot rust">.</span>state <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="meta path rust">State<span class="punctuation accessor rust">::</span></span>Halted <span class="keyword operator rust">=></span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="variable language rust">self</span><span class="punctuation accessor dot rust">.</span>fib<span class="punctuation accessor dot rust">.</span>state <span class="keyword operator assignment rust">=</span> <span class="meta path rust">State<span class="punctuation accessor rust">::</span></span>Running<span class="punctuation terminator rust">;</span>
<span class="meta path rust">Poll<span class="punctuation accessor rust">::</span></span>Ready<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
<span class="meta path rust">State<span class="punctuation accessor rust">::</span></span>Running <span class="keyword operator rust">=></span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="variable language rust">self</span><span class="punctuation accessor dot rust">.</span>fib<span class="punctuation accessor dot rust">.</span>state <span class="keyword operator assignment rust">=</span> <span class="meta path rust">State<span class="punctuation accessor rust">::</span></span>Halted<span class="punctuation terminator rust">;</span>
<span class="meta path rust">Poll<span class="punctuation accessor rust">::</span></span>Pending
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div><h2 id="executor">Executor</h2>
<p>Our executor keeps a vector of uncompleted futures, where the state of each
future is located on the heap. As a very basic executor, it only supports
adding futures to it before actual execution takes place and not afterward.
The <code>push</code> method adds a closure to the list of futures, and the <code>run</code> method
performs interlaced execution of futures until all of them complete.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="keyword other rust">use</span> <span class="meta path rust">std<span class="punctuation accessor rust">::</span></span><span class="meta path rust">collections<span class="punctuation accessor rust">::</span></span>VecDeque<span class="punctuation terminator rust">;</span>
<span class="meta struct rust"><span class="storage type struct rust">struct</span> </span><span class="meta struct rust"><span class="entity name struct rust">Executor</span> </span><span class="meta struct rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="variable other member rust">fibs</span><span class="punctuation separator type rust">:</span> <span class="meta generic rust">VecDeque<span class="punctuation definition generic begin rust"><</span><span class="meta generic rust">Pin<span class="punctuation definition generic begin rust"><</span><span class="meta generic rust"><span class="support type rust">Box</span><span class="punctuation definition generic begin rust"><</span>dyn <span class="meta generic rust">Future<span class="punctuation definition generic begin rust"><</span>Output=<span class="punctuation section group begin rust">(</span><span class="punctuation section group end rust">)</span><span class="punctuation definition generic end rust">></span></span><span class="punctuation definition generic end rust">></span></span><span class="punctuation definition generic end rust">></span></span><span class="punctuation definition generic end rust">></span></span>,
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta impl rust"><span class="storage type impl rust">impl</span> </span><span class="meta impl rust"><span class="entity name impl rust">Executor</span> </span><span class="meta impl rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">new</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> <span class="storage type rust">Self</span></span> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
Executor <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
fibs<span class="punctuation separator rust">:</span> <span class="meta path rust">VecDeque<span class="punctuation accessor rust">::</span></span>new<span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation separator rust">,</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">push</span></span><span class="meta generic rust"><span class="punctuation definition generic begin rust"><</span>C, F<span class="punctuation definition generic end rust">></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">&</span><span class="storage modifier rust">mut</span> <span class="variable parameter rust">self</span>, <span class="variable parameter rust">closure</span><span class="punctuation separator rust">:</span> C</span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust">
</span><span class="meta function rust"><span class="keyword other rust">where</span>
F<span class="punctuation separator rust">:</span> <span class="meta generic rust">Future<span class="punctuation definition generic begin rust"><</span>Output=<span class="punctuation section group begin rust">(</span><span class="punctuation section group end rust">)</span><span class="punctuation definition generic end rust">></span></span> + <span class="storage modifier lifetime rust">'static</span>,
C<span class="punctuation separator rust">:</span> FnOnce<span class="punctuation section group begin rust">(</span>Fib<span class="punctuation section group end rust">)</span> -> F,
</span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type rust">let</span> fib <span class="keyword operator assignment rust">=</span> Fib <span class="meta block rust"><span class="punctuation section block begin rust">{</span> state<span class="punctuation separator rust">:</span> <span class="meta path rust">State<span class="punctuation accessor rust">::</span></span>Running </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span><span class="punctuation terminator rust">;</span>
<span class="variable language rust">self</span><span class="punctuation accessor dot rust">.</span>fibs<span class="punctuation accessor dot rust">.</span><span class="support function rust">push_back</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="support type rust">Box</span><span class="meta path rust"><span class="punctuation accessor rust">::</span></span>pin<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="support function rust">closure</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span>fib</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">run</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">&</span><span class="storage modifier rust">mut</span> <span class="variable parameter rust">self</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type rust">let</span> waker <span class="keyword operator assignment rust">=</span> <span class="meta path rust">waker<span class="punctuation accessor rust">::</span></span>create<span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="storage type rust">let</span> <span class="storage modifier rust">mut</span> context <span class="keyword operator assignment rust">=</span> <span class="meta path rust">Context<span class="punctuation accessor rust">::</span></span>from_waker<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="keyword operator bitwise rust">&</span>waker</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="keyword control rust">while</span> <span class="storage type rust">let</span> <span class="support type rust">Some</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="storage modifier rust">mut</span> fib</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span> <span class="keyword operator assignment rust">=</span> <span class="variable language rust">self</span><span class="punctuation accessor dot rust">.</span>fibs<span class="punctuation accessor dot rust">.</span><span class="support function rust">pop_front</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="keyword control rust">match</span> fib<span class="punctuation accessor dot rust">.</span><span class="support function rust">as_mut</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation accessor dot rust">.</span><span class="support function rust">poll</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="keyword operator bitwise rust">&</span><span class="storage modifier rust">mut</span> context</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="meta path rust">Poll<span class="punctuation accessor rust">::</span></span>Pending <span class="keyword operator rust">=></span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="variable language rust">self</span><span class="punctuation accessor dot rust">.</span>fibs<span class="punctuation accessor dot rust">.</span><span class="support function rust">push_back</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span>fib</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span><span class="punctuation separator rust">,</span>
<span class="meta path rust">Poll<span class="punctuation accessor rust">::</span></span>Ready<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span> <span class="keyword operator rust">=></span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span></span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span><span class="punctuation separator rust">,</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div><h2 id="null-waker">Null Waker</h2>
<p>For the executor implementation above, we need a null
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://docs.rs/futures/0.3.1/futures/task/struct.Waker.html">Waker<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, similar
to the one used in <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/whatisaphone/genawaiter">genawaiter<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>
(<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/whatisaphone/genawaiter/blob/master/src/waker.rs">link<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>).</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="keyword other rust">use</span> <span class="meta path rust">std<span class="punctuation accessor rust">::</span></span><span class="meta path rust">task<span class="punctuation accessor rust">::</span></span><span class="meta block rust"><span class="punctuation section block begin rust">{</span>RawWaker<span class="punctuation separator rust">,</span> RawWakerVTable<span class="punctuation separator rust">,</span> Waker</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span><span class="punctuation separator rust">,</span>
<span class="meta function rust"><span class="meta function rust"><span class="storage modifier rust">pub</span> <span class="storage type function rust">fn</span> </span><span class="entity name function rust">create</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> Waker</span> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="comment line double-slash rust"><span class="punctuation definition comment rust">//</span> Safety: The waker points to a vtable with functions that do nothing. Doing
</span> <span class="comment line double-slash rust"><span class="punctuation definition comment rust">//</span> nothing is memory-safe.
</span> <span class="storage modifier rust">unsafe</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span> <span class="meta path rust">Waker<span class="punctuation accessor rust">::</span></span>from_raw<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="constant other rust">RAW_WAKER</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span> </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="storage type rust">const</span> <span class="constant other rust">RAW_WAKER</span><span class="punctuation separator rust">:</span> RawWaker <span class="keyword operator assignment rust">=</span> <span class="meta path rust">RawWaker<span class="punctuation accessor rust">::</span></span>new<span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="meta path rust">std<span class="punctuation accessor rust">::</span></span><span class="meta path rust">ptr<span class="punctuation accessor rust">::</span></span>null<span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation separator rust">,</span> <span class="keyword operator bitwise rust">&</span><span class="constant other rust">VTABLE</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="storage type rust">const</span> <span class="constant other rust">VTABLE</span><span class="punctuation separator rust">:</span> RawWakerVTable <span class="keyword operator assignment rust">=</span> <span class="meta path rust">RawWakerVTable<span class="punctuation accessor rust">::</span></span>new<span class="meta group rust"><span class="punctuation section group begin rust">(</span>clone<span class="punctuation separator rust">,</span> wake<span class="punctuation separator rust">,</span> wake_by_ref<span class="punctuation separator rust">,</span> drop</span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="storage modifier rust">unsafe</span> <span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">clone</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">_</span>: <span class="storage modifier rust">*const</span> <span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> <span class="meta function return-type rust"><span class="punctuation separator rust">-></span> RawWaker</span> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span> <span class="constant other rust">RAW_WAKER</span> </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="storage modifier rust">unsafe</span> <span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">wake</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">_</span>: <span class="storage modifier rust">*const</span> <span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span> </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="storage modifier rust">unsafe</span> <span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">wake_by_ref</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">_</span>: <span class="storage modifier rust">*const</span> <span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span> </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
<span class="storage modifier rust">unsafe</span> <span class="meta function rust"><span class="meta function rust"><span class="storage type function rust">fn</span> </span><span class="entity name function rust">drop</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span><span class="keyword operator rust">_</span>: <span class="storage modifier rust">*const</span> <span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span> </span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div><h2 id="giving-it-a-go">Giving it a go</h2>
<p>We can test the library using a program such as the following:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="meta function rust"><span class="meta function rust"><span class="storage modifier rust">pub</span> <span class="storage type function rust">fn</span> </span><span class="entity name function rust">main</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type rust">let</span> <span class="storage modifier rust">mut</span> exec <span class="keyword operator assignment rust">=</span> <span class="meta path rust">Executor<span class="punctuation accessor rust">::</span></span>new<span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="keyword control rust">for</span> instance <span class="keyword operator rust">in</span> <span class="constant numeric integer decimal rust">1</span><span class="keyword operator range rust">..</span><span class="keyword operator assignment rust">=</span><span class="constant numeric integer decimal rust">3</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
exec<span class="punctuation accessor dot rust">.</span><span class="support function rust">push</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="storage modifier rust">move</span> <span class="keyword operator bitwise rust">|</span><span class="storage modifier rust">mut</span> fib<span class="keyword operator bitwise rust">|</span> async <span class="storage modifier rust">move</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span><span class="constant other placeholder rust">{}</span> A<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation separator rust">,</span> instance<span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
fib<span class="punctuation accessor dot rust">.</span><span class="support function rust">waiter</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation accessor dot rust">.</span>await<span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span><span class="constant other placeholder rust">{}</span> B<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation separator rust">,</span> instance<span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
fib<span class="punctuation accessor dot rust">.</span><span class="support function rust">waiter</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation accessor dot rust">.</span>await<span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span><span class="constant other placeholder rust">{}</span> C<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation separator rust">,</span> instance<span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
fib<span class="punctuation accessor dot rust">.</span><span class="support function rust">waiter</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation accessor dot rust">.</span>await<span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span><span class="constant other placeholder rust">{}</span> D<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation separator rust">,</span> instance<span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Running<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
exec<span class="punctuation accessor dot rust">.</span><span class="support function rust">run</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Done<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div>
<p>The output is as follows:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">Running
1 A
2 A
3 A
1 B
2 B
3 B
1 C
2 C
3 C
1 D
2 D
3 D
Done
</span></td></pre></tr></tbody></table></div><h2 id="performance">Performance</h2>
<p>Timing the following program compiled with <code>lto = true</code>, I have seen that it
takes about 5 nanoseconds for each iteration of the internal loop, on an Intel
i7-7820X CPU.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source rust"><span class="meta function rust"><span class="meta function rust"><span class="storage modifier rust">pub</span> <span class="storage type function rust">fn</span> </span><span class="entity name function rust">bench</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters begin rust">(</span></span><span class="meta function rust"><span class="meta function parameters rust"><span class="punctuation section parameters end rust">)</span></span></span></span><span class="meta function rust"> </span><span class="meta function rust"><span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="storage type rust">let</span> <span class="storage modifier rust">mut</span> exec <span class="keyword operator assignment rust">=</span> <span class="meta path rust">Executor<span class="punctuation accessor rust">::</span></span>new<span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="keyword control rust">for</span> <span class="keyword operator rust">_</span> <span class="keyword operator rust">in</span> <span class="constant numeric integer decimal rust">1</span><span class="keyword operator range rust">..</span><span class="keyword operator assignment rust">=</span><span class="constant numeric integer decimal rust">2</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
exec<span class="punctuation accessor dot rust">.</span><span class="support function rust">push</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span><span class="storage modifier rust">move</span> <span class="keyword operator bitwise rust">|</span><span class="storage modifier rust">mut</span> fib<span class="keyword operator bitwise rust">|</span> async <span class="storage modifier rust">move</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
<span class="keyword control rust">for</span> <span class="keyword operator rust">_</span> <span class="keyword operator rust">in</span> <span class="constant numeric integer decimal rust">0</span><span class="keyword operator range rust">..</span><span class="constant numeric integer decimal rust">100_000_000</span> <span class="meta block rust"><span class="punctuation section block begin rust">{</span>
fib<span class="punctuation accessor dot rust">.</span><span class="support function rust">waiter</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation accessor dot rust">.</span>await<span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Running<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
exec<span class="punctuation accessor dot rust">.</span><span class="support function rust">run</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
<span class="support macro rust">println!</span><span class="meta group rust"><span class="punctuation section group begin rust">(</span></span><span class="meta group rust"><span class="string quoted double rust"><span class="punctuation definition string begin rust">"</span>Done<span class="punctuation definition string end rust">"</span></span></span><span class="meta group rust"><span class="punctuation section group end rust">)</span></span><span class="punctuation terminator rust">;</span>
</span><span class="meta block rust"><span class="punctuation section block end rust">}</span></span></span>
</span></td></pre></tr></tbody></table></div><h2 id="end-notes">End notes</h2>
<p>One of the nice things about the <code>async</code>/<code>await</code> support in the Rust compiler
is that it does not depend on any specific run-time. Thus, if you commit to
certain run-time, you are free to implement your own executor.</p>
<p>Independency of run-time has its downsides. For example, the library presented
here is not compatible with other run-times such as
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://docs.rs/async-std"><code>async-std</code><span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>. And in fact, the implementation
violates the interface intended for the <code>Future</code>'s <code>poll</code> function by assuming
that the <code>Future</code> will always be <code>Ready</code> after it was <code>Pending</code>.</p>
<p>Combined uses of several run-times in a single program is possible but requires
extra care (see <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.reddit.com/r/rust/comments/eagjyf/using_libraries_depending_on_different_async/">Reddit
discussion<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>).</p>
How to Easily Patch Fedora Packages2020-01-14T00:00:00+00:002020-01-14T00:00:00+00:00https://blog.aloni.org/posts/how-to-easily-patch-fedora-packages/<p>While many Linux users are developers, a rather small part of the developers
feels easy to meddle or extend their distribution's packaging. When there is a
pending feature from an upstream package, or when there is a bug in a package,
users usually wait for the distribution maintainers to fix it, or resort to use
independent solutions such as
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://docs.brew.sh/Homebrew-on-Linux">homebrew<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, for rebuilding upstream
projects.</p>
<p>Software packaging in distributions entails dealing with the build system of an
unknown subproject, and it is sometimes a learning curve that may deter
developers from tackling it. Luckily, in <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://getfedora.org/">Fedora
Linux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, we have good tools allowing to deal each
package's own mess, and make it surprisingly easy to build modified versions.</p>
<span id="continue-reading"></span><h2 id="obtaining-the-source">Obtaining the source</h2>
<p>First, we begin by installing several base system package that will assist us
in dealing with package building:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ sudo dnf install fedpkg make
</span></td></pre></tr></tbody></table></div>
<p>Let's say we want to consider a patch or a feature for <code>zsh</code> (<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Z_shell">Z
shell<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>).</p>
<p>We need to consider what is the source package project from which the resultant
binaries were made. It's easy to query the <code>rpm</code> tool for that. Based on a
filename of an installed package, the name of the source package RPM can be
revealed. For example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ rpm -qif /usr/bin/zsh | grep 'Source RPM'
Source RPM : zsh-5.7.1-4.fc30.src.rpm
</span></td></pre></tr></tbody></table></div>
<p>It is not surprising that the source RPM name is <code>zsh</code> too. It may not be the
case for other packages though.</p>
<p>Normally, the source package RPM name matches the name of the Git repository in
which Fedora maintains the scripts that allow building it. Cloning a source
package from Fedora's Git server is very easy, and <strong>does not</strong> require
becoming a Fedora community member or any other credential. For <code>zsh</code>, we can
perform the following command using <code>fedpkg</code>:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ fedpkg co -a zsh
Cloning into 'zsh'...
remote: Counting objects: 1023, done.
remote: Compressing objects: 100% (777/777), done.
remote: Total 1023 (delta 556), reused 423 (delta 216)
Receiving objects: 100% (1023/1023), 235.29 KiB | 271.00 KiB/s, done.
Resolving deltas: 100% (556/556), done.
</span></td></pre></tr></tbody></table></div>
<p>For each package, the source for each version of Fedora is maintained in a
different branch. We can therefore check-out the branch that is matching the
version of the distribution for which we are building:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ cd zsh
$ git checkout -b f31 origin/f31
Branch 'f31' set up to track remote branch 'f31' from 'origin'.
Switched to a new branch 'f31'
</span></td></pre></tr></tbody></table></div>
<p>From now on we will invoke <code>fedpkg</code> in the working directory of the package.</p>
<h2 id="setting-up-a-build-environment">Setting up a build environment</h2>
<p>Different packages will have varying requirements for a functioning development
environment. Luckily, Fedora's <code>dnf</code> assists us in bringing in the dependencies
needed for any of the packages that it can build. This can be done via the <code>dnf builddep</code> command. For our use case, we can bring in <code>zsh</code> dependencies:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ sudo dnf builddep zsh
</span></td></pre></tr></tbody></table></div><h2 id="directly-building-a-distributable-rpm">Directly building a distributable RPM</h2>
<p>Before modifying the package, it is worth testing to see if we are able to
build it correctly even without any modification. The following command will
try to build the binary RPMs from the current state of the code:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ fedpkg local
</span></td></pre></tr></tbody></table></div>
<p>We can observe that the following RPMs have been generated:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ ls -l1 x86_64/ noarch/
noarch/:
total 452
-rw-rw-r--. 1 user user 459748 Jan 14 13:58 zsh-html-5.7.1-4.fc31.noarch.rpm
x86_64/:
total 5476
-rw-rw-r--. 1 user user 2999294 Jan 14 13:58 zsh-5.7.1-4.fc31.x86_64.rpm
-rw-rw-r--. 1 user user 1771784 Jan 14 13:58 zsh-debuginfo-5.7.1-4.fc31.x86_64.rpm
-rw-rw-r--. 1 user user 829306 Jan 14 13:58 zsh-debugsource-5.7.1-4.fc31.x86_64.rpm
</span></td></pre></tr></tbody></table></div><h2 id="alternatively-building-the-rpm-in-a-mock-container">Alternatively, building the RPM in a <code>mock</code> container</h2>
<p>Even before docker containers were popular, Fedora provided us with a tool named <code>mock</code>
that creates a separate environment for building packages. Thus, it can be
used to build the package independently of the development environment.</p>
<p>First, we need to make sure that <code>mock</code> is installed.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ sudo dnf install mock
</span></td></pre></tr></tbody></table></div>
<p>Then, we can tell <code>fedpkg</code> to use <code>mock</code> in order to build the package:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ fedpkg mockbuild
</span></td></pre></tr></tbody></table></div>
<p>The build outputs of <code>mock</code> are all moved to a directory containing the log
files of the build, and the version and name of the package:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ ls -lR results_zsh/*/*
results_zsh/5.7.1/4.fc31:
total 9852
-rw-rw-r--. 1 user user 190779 Jan 14 14:10 build.log
-rw-rw-r--. 1 user user 2744 Jan 14 14:03 hw_info.log
-rw-rw-r--. 1 user user 52642 Jan 14 14:08 installed_pkgs.log
-rw-rw-r--. 1 user user 614047 Jan 14 14:10 root.log
-rw-rw-r--. 1 user user 998 Jan 14 14:10 state.log
-rw-r--r--. 1 user mock 3146067 Jan 14 14:06 zsh-5.7.1-4.fc31.src.rpm
-rw-r--r--. 1 user mock 2999346 Jan 14 14:10 zsh-5.7.1-4.fc31.x86_64.rpm
-rw-r--r--. 1 user mock 1772488 Jan 14 14:10 zsh-debuginfo-5.7.1-4.fc31.x86_64.rpm
-rw-r--r--. 1 user mock 829174 Jan 14 14:10 zsh-debugsource-5.7.1-4.fc31.x86_64.rpm
-rw-r--r--. 1 user mock 459682 Jan 14 14:10 zsh-html-5.7.1-4.fc31.noarch.rpm
</span></td></pre></tr></tbody></table></div>
<p>Using <code>mock</code> to perform the build has the advantage in so that it verifies that
the dependencies of the build are properly specified, and it can also be used
to build for different versions of the distribution on the same machine. It's
also the working horse behind Fedora's own build servers, and namely
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://copr.fedorainfracloud.org">Copr<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<h2 id="adding-a-patch">Adding a patch</h2>
<p>To produce a patch for a package, we need the source of the package itself,
rather than the sources for the Fedora scripts that tell how to build it.</p>
<p>There are several ways to obtain the source, and one of them is by using
<code>fedpkg</code>. We can tell it to create a directory containing the patched sources
of the package, as they are ready to be built. This process will execute the
<code>prep</code> stage of the RPM spec source, and the result will usually be a directory
directly under our working tree.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ fedpkg prep
</span></td></pre></tr></tbody></table></div>
<p>Because Fedora source packages do not assume about the source control aspects
of upstream projects, they contain just archives of a certain version's source.
The created directory is not tracked in source control, and if we want to
modify it, usually it is best to move it aside to a different directory, and
initialize it with Git. For example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ mv zsh-5.7.1 ../zsh-5.7.1
$ cd ../zsh-5.7.1
$ git init && git add -f . && git commit -m "Base version"
</span></td></pre></tr></tbody></table></div>
<p>Here, our <code>zsh-5.7.1</code> is only a representation of that Fedora-maintained
version, that is probably already patched by Fedora to some degree, but it can
and should be used as a base for our further patching. Though, we may want to
have a clone of the upstream project, <code>zsh</code> in that case, handy for full Git
history browsing.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ cd ..
$ git clone git://git.code.sf.net/p/zsh/code zsh-upstream
Cloning into 'zsh-upstream'...
remote: Enumerating objects: 94026, done.
remote: Counting objects: 100% (94026/94026), done.
remote: Compressing objects: 100% (25128/25128), done.
remote: Total 94026 (delta 73505), reused 87930 (delta 68487)
Receiving objects: 100% (94026/94026), 16.60 MiB | 1.07 MiB/s, done.
Resolving deltas: 100% (73505/73505), done.
</span></td></pre></tr></tbody></table></div>
<p>Going back to either of our source clones we can proceed to committing changes and
generating patches from commits. Here is an example for a trivial patch:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source diff">$ git diff HEAD
diff --git a/Src/hist.c b/Src/hist.c
index dbdc1e4..cdb1dd1 100644
<span class="meta diff header from-file"><span class="meta header from-file diff"><span class="punctuation definition from-file diff">---</span> a/Src/hist.c
</span></span><span class="meta diff header to-file"><span class="meta header to-file diff"><span class="punctuation definition to-file diff">+++</span> b/Src/hist.c
</span></span><span class="meta diff range unified"><span class="meta range unified diff"><span class="punctuation definition range diff">@@</span> <span class="meta toc-list line-number diff">-580,7 +580,7</span> <span class="punctuation definition range diff">@@</span> <span class="entity name section diff">histsubchar(int c)</span>
</span></span> */
lexraw_mark = zshlex_raw_mark(-1);
<span class="markup deleted diff"><span class="punctuation definition deleted diff">-</span> /* look, no goto's */
</span><span class="markup inserted diff"><span class="punctuation definition inserted diff">+</span> /* look, no goto's! */
</span> if (isfirstch && c == hatchar) {
int gbal = 0;
$ git commit -m "Adding an exclamation mark to Src/hist.c"
[f31 c00d68b] Adding an exclamation mark to Src/hist.c
2 files changed, 100 insertions(+), 1 deletion(-)
create mode 100644 0001-zsh-5.7.1-zle-history-avoid-crash.patch
$ git format-patch HEAD~1 -o ../zsh
../zsh/0001-Adding-an-exclamation-mark-to-Src-hist.c.patch
</span></td></pre></tr></tbody></table></div>
<p>The <code>git format-patch</code> is a handy command that exports Git commits as files,
and its output can be used as input to the package building process, as the
packaging standard in Fedora mandates separation of the upstream sources from
the patches that were made on them.</p>
<p>Adding the patch files to the Fedora package's source specification may be a
bit tricky, but after a few times you understand that the packaging format is
simpler than it may seem at first. In the case of <code>zsh</code>, we only need to specify
the newly created patch in a <code>Patch<number>: </code> line in the beginning of <code>zsh.spec</code>.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source diff">$ git diff
diff --git a/zsh.spec b/zsh.spec
index 0d77f70..0022a90 100644
<span class="meta diff header from-file"><span class="meta header from-file diff"><span class="punctuation definition from-file diff">---</span> a/zsh.spec
</span></span><span class="meta diff header to-file"><span class="meta header to-file diff"><span class="punctuation definition to-file diff">+++</span> b/zsh.spec
</span></span><span class="meta diff range unified"><span class="meta range unified diff"><span class="punctuation definition range diff">@@</span> <span class="meta toc-list line-number diff">-14,6 +14,7</span> <span class="punctuation definition range diff">@@</span> <span class="entity name section diff">Source6: dotzshrc</span>
</span></span>
# make failed searches of history in Zle robust (#1722703)
Patch1: 0001-zsh-5.7.1-zle-history-avoid-crash.patch
<span class="markup inserted diff"><span class="punctuation definition inserted diff">+</span>Patch2: 0001-Adding-an-exclamation-mark-to-Src-hist.c.patch
</span>
BuildRequires: autoconf
BuildRequires: coreutils
</span></td></pre></tr></tbody></table></div>
<p>Older packages may require some more changes, such as adding extra <code>%patch</code>
lines further in the file, relating to the aforementioned <code>prep</code> stage.</p>
<h2 id="identifying-a-patched-package">Identifying a patched package</h2>
<p>If no other <code>.spec</code> fields are changed, our modified package would appear
mostly indistinguishable in meta-data from the original package. Usually this
is not wanted. Therefore, it is best to modify the <code>.spec</code> to include a string
in the package's <code>Release</code> field. For example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source diff">diff --git a/zsh.spec b/zsh.spec
index 0022a90..78c362e 100644
<span class="meta diff header from-file"><span class="meta header from-file diff"><span class="punctuation definition from-file diff">---</span> a/zsh.spec
</span></span><span class="meta diff header to-file"><span class="meta header to-file diff"><span class="punctuation definition to-file diff">+++</span> b/zsh.spec
</span></span><span class="meta diff range unified"><span class="meta range unified diff"><span class="punctuation definition range diff">@@</span> <span class="meta toc-list line-number diff">-1,7 +1,7</span> <span class="punctuation definition range diff">@@</span>
</span></span> Summary: Powerful interactive shell
Name: zsh
Version: 5.7.1
<span class="markup deleted diff"><span class="punctuation definition deleted diff">-</span>Release: 4%{?dist}
</span><span class="markup inserted diff"><span class="punctuation definition inserted diff">+</span>Release: 4%{?dist}.daloni
</span> License: MIT
URL: http://zsh.sourceforge.net/
Source0: https://downloads.sourceforge.net/%{name}/%{name}-%{version}.tar.xz
</span></td></pre></tr></tbody></table></div>
<p>We can see that following a build, the <code><name>-<version>-<release></code> triplet is extended:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ ls -l1 x86_64/ noarch/
noarch/:
total 452
-rw-rw-r--. 1 user user 459790 Jan 14 15:15 zsh-html-5.7.1-4.fc31.<span class="sourceMarker">daloni</span>.noarch.rpm
x86_64/:
total 5476
-rw-rw-r--. 1 user user 2999482 Jan 14 15:15 zsh-5.7.1-4.fc31.<span class="sourceMarker">daloni</span>.x86_64.rpm
-rw-rw-r--. 1 user user 1773094 Jan 14 15:15 zsh-debuginfo-5.7.1-4.fc31.<span class="sourceMarker">daloni</span>.x86_64.rpm
-rw-rw-r--. 1 user user 829124 Jan 14 15:15 zsh-debugsource-5.7.1-4.fc31.<span class="sourceMarker">daloni</span>.x86_64.rpm
</span></td></pre></tr></tbody></table></div><h2 id="installing-the-package">Installing the package</h2>
<p>The new packages can be installed via <code>dnf install</code>. Once the patched packages
are installed, it is easy to spot them by <code>grep</code>-ing over the output of <code>dnf list installed</code> by various means.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ sudo dnf install x86_64/zsh-5.7.1-4.fc31.daloni.x86_64.rpm
$ dnf list installed | grep @@commandline
zsh.x86_64 5.7.1-4.fc31.daloni @@commandline
$ dnf list installed | grep daloni
zsh.x86_64 5.7.1-4.fc31.daloni @@commandline
</span></td></pre></tr></tbody></table></div><h2 id="avoiding-accidental-overrides-from-a-newer-package-version">Avoiding accidental overrides from a newer package version</h2>
<p>When Fedora releases newer version of the package, automatic system upgrades
may override the patched one due to providing a higher version. There are
several ways to prevent this issue, but the one I prefer is excluding upgrades
on such packages from <code>dnf</code>'s configuration:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ grep exclude /etc/dnf/dnf.conf
exclude=zsh
</span></td></pre></tr></tbody></table></div>
<p>Furthermore, there are various aspects of dealing with packages that are
outside the scope of this post. An exercise left for the reader is to create an
RPM repository containing the patched package, host it on the network, and for
client machines - devise a repository description file to be placed in
<code>/etc/yum.repos.d</code>, and configure it to have a higher priority than the
original distribution packages.</p>
<h2 id="keep-the-debuginfo">Keep the debuginfo!</h2>
<p>If a patched package has crashed and generated a corefile, the special
debuginfo packages that were produced in the build may be handy in examining
this corefile. Therefore, you may want to keep these debuginfo packages,
especially because a reproduced build is sometimes difficult to match in
complete binary compatibility to the original one. This depends on how
deterministically the package is being built, a matter that is intrinsic to the
package's own build system.</p>
Computing Symbolic Gradient Vectors with Plain Haskell2016-04-27T00:00:00+00:002016-04-27T00:00:00+00:00https://blog.aloni.org/posts/symbolic-gradients-with-plain-haskell/<p>While writing my <a href="/posts/backprop-with-tensorflow">previous post</a>, I was curious how easy it would be to implement <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.tensorflow.org">TensorFlow<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>'s automatic differentiation for back propagation. In TensorFlow's web site they call it 'automatic differentiation' but in fact they probably do 'symbolic differentiation', as <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://download.tensorflow.org/paper/whitepaper2015.pdf">mentioned in their white paper<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>. The difference between the two relates to whether the differentiation is done during the original computation or beforehand. It makes sense to do the latter, because then you can maintain a separate computational graph of the back propagation to perform the updates.</p>
<span id="continue-reading"></span>
<p>I've looked into this topic in the <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Haskell_(programming_language)">Haskell<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> ecosystem, and found many useful and extensive libraries, namely <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://hackage.haskell.org/package/ad">ad<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> by Edward Kmett. However, to use these libraries or understand many of the blog posts on the subject requires some advanced Haskell, and I was wondering whether one can get going with differentiation using very basic and lean use of Haskell.</p>
<p><em>So, how easy it would be to compute gradients of single-output functions, using Haskell with only the basic arsenal at our hands?</em></p>
<h2 id="data-and-imports">Data and imports</h2>
<p>First, we perform some imports and declare the basic data type to hold our expression tree:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta import haskell"><span class="keyword other haskell">import</span> <span class="support other module haskell">Control.Monad</span> <span class="meta declaration exports haskell">(<span class="entity name function haskell">forM_</span>)</span></span>
<span class="meta import haskell"><span class="keyword other haskell">import</span> <span class="keyword other haskell">qualified</span> <span class="support other module haskell">Data.Map</span> <span class="keyword other haskell">as</span> <span class="support other module haskell">Map</span></span>
<span class="keyword other haskell">data</span> <span class="constant other haskell">Expr</span>
<span class="keyword operator haskell">=</span> <span class="constant other haskell">Term</span> <span class="constant other haskell">Int</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> 'Term 0' is x0, 'Term 1' is x1, etc..
</span> <span class="keyword operator haskell">|</span> <span class="constant other haskell">Lit</span> <span class="constant other haskell">Float</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> Constant numbers
</span> <span class="keyword operator haskell">|</span> <span class="constant other haskell">Neg</span> <span class="constant other haskell">Expr</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> -f
</span> <span class="keyword operator haskell">|</span> <span class="constant other haskell">Mul</span> <span class="constant other haskell">Expr</span> <span class="constant other haskell">Expr</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> a + b
</span> <span class="keyword operator haskell">|</span> <span class="constant other haskell">Add</span> <span class="constant other haskell">Expr</span> <span class="constant other haskell">Expr</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> a * b
</span> <span class="keyword operator haskell">|</span> <span class="constant other haskell">Sin</span> <span class="constant other haskell">Expr</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> sin a
</span> <span class="keyword operator haskell">|</span> <span class="constant other haskell">Cos</span> <span class="constant other haskell">Expr</span> <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> cos a
</span> <span class="meta deriving haskell"><span class="keyword other haskell">deriving</span> (<span class="entity other inherited-class haskell">Show</span>, <span class="entity other inherited-class haskell">Eq</span>, <span class="entity other inherited-class haskell">Ord</span>)</span>
</span></td></pre></tr></tbody></table></div>
<p>We shall be able to derive symbolic gradients for any function built with this data type.</p>
<h2 id="poor-man-s-pretty-printing">Poor man's pretty-printing</h2>
<p>One cannot go by without a nice String representation:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">fshow</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Expr</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">String</span>
</span>fshow (<span class="constant other haskell">Term</span> v) <span class="keyword operator haskell">=</span> concat [<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>x<span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> show v]
fshow (<span class="constant other haskell">Lit</span> v) <span class="keyword operator haskell">=</span> show v
fshow (<span class="constant other haskell">Mul</span> e1 e2) <span class="keyword operator haskell">=</span> concat [<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>(<span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e1<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span> * <span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e2<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>)<span class="punctuation definition string end haskell">"</span></span>]
fshow (<span class="constant other haskell">Add</span> e1 e2) <span class="keyword operator haskell">=</span> concat [<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>(<span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e1<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span> + <span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e2<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>)<span class="punctuation definition string end haskell">"</span></span>]
fshow (<span class="constant other haskell">Neg</span> e) <span class="keyword operator haskell">=</span> concat [<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>-(<span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>)<span class="punctuation definition string end haskell">"</span></span>]
fshow (<span class="constant other haskell">Sin</span> e) <span class="keyword operator haskell">=</span> concat [<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>sin(<span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>)<span class="punctuation definition string end haskell">"</span></span>]
fshow (<span class="constant other haskell">Cos</span> e) <span class="keyword operator haskell">=</span> concat [<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>cos(<span class="punctuation definition string end haskell">"</span></span><span class="punctuation separator comma haskell">,</span> fshow e<span class="punctuation separator comma haskell">,</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>)<span class="punctuation definition string end haskell">"</span></span>]
</span></td></pre></tr></tbody></table></div>
<p>This implementation is basic in so that a sequence of summations will bear a horrible representation similar to <code>(x1 + (x2 + (x3 + (...))))</code> - however it's enough to get us going.</p>
<h2 id="sample">Sample</h2>
<p>The <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Automatic_differentiation">Wikipedia page for Automatic Differentiation<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> demonstrates with the following function:</p>
<p>$$
f(x_1, x_2) = \sin x_1 + x_1x_2
$$</p>
<p>It should be easy enough to represent it with our Haskell data, and use <code>fshow</code> from above:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell">λ<span class="keyword operator haskell">></span> <span class="keyword other haskell">let</span> wikipediaFunc <span class="keyword operator haskell">=</span> (<span class="constant other haskell">Sin</span> (<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">1</span>)) <span class="keyword operator function infix haskell"><span class="punctuation definition entity haskell">`</span>Add<span class="punctuation definition entity haskell">`</span></span> ((<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">1</span>) <span class="keyword operator function infix haskell"><span class="punctuation definition entity haskell">`</span>Mul<span class="punctuation definition entity haskell">`</span></span> (<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">2</span>))
λ<span class="keyword operator haskell">></span> fshow wikipediaFunc
<span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>(sin(x1) + (x1 * x2))<span class="punctuation definition string end haskell">"</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="gradient">Gradient</h2>
<p>The <code>gradient</code> function below takes an expression, and returns a map from each term number to the expression that computes it. The definition of the function is recursive and based on known simple derivation rules:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">gradient</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Expr</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">Map</span>.<span class="storage type haskell">Map</span> <span class="storage type haskell">Int</span> <span class="storage type haskell">Expr</span>
</span>gradient (<span class="constant other haskell">Neg</span> e) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>map <span class="constant other haskell">Neg</span> (gradient e)
gradient (<span class="constant other haskell">Cos</span> e) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>map (<span class="constant other haskell">Mul</span> (<span class="constant other haskell">Neg</span> (<span class="constant other haskell">Sin</span> e))) (gradient e)
gradient (<span class="constant other haskell">Sin</span> e) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>map (<span class="constant other haskell">Mul</span> (<span class="constant other haskell">Cos</span> e)) (gradient e)
gradient (<span class="constant other haskell">Term</span> i) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>fromList [(i<span class="punctuation separator comma haskell">,</span> <span class="constant other haskell">Lit</span> <span class="constant numeric float decimal haskell">1<span class="punctuation separator decimal haskell">.</span>0</span>)]
gradient (<span class="constant other haskell">Lit</span> _) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>empty
gradient (<span class="constant other haskell">Add</span> e1 e2) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>unionWith <span class="constant other haskell">Add</span> (gradient e1) (gradient e2)
gradient (<span class="constant other haskell">Mul</span> e1 e2) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>unionWith <span class="constant other haskell">Add</span> (<span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>map (<span class="constant other haskell">Mul</span> e2) (gradient e1))
(<span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>map (<span class="constant other haskell">Mul</span> e1) (gradient e2))
</span></td></pre></tr></tbody></table></div>
<p>The interesting parts are where <code>Map.unionWith</code> is used for addition and multiplication. Notice how easily the <code>Mul</code> part relates to the known derivation rule:</p>
<p>$$(f(x)g(x))' = g(x)f'(x) + g'(x)f(x)$$</p>
<p>The documentation for <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://hackage.haskell.org/package/containers/docs/Data-Map-Strict.html">Data.Map<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> can tell about <code>Map.map</code> and <code>Map.unionWith</code>.</p>
<h2 id="small-helpers">Small helpers</h2>
<p>Before testing it, we'll add just two helper functions. The first function simplifies expressions by getting rid of the $1.0$ literals we have added.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">simplify</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Expr</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">Expr</span>
</span>simplify (<span class="constant other haskell">Mul</span> (<span class="constant other haskell">Lit</span> <span class="constant numeric float decimal haskell">1<span class="punctuation separator decimal haskell">.</span>0</span>) e) <span class="keyword operator haskell">=</span> simplify e
simplify (<span class="constant other haskell">Mul</span> e (<span class="constant other haskell">Lit</span> <span class="constant numeric float decimal haskell">1<span class="punctuation separator decimal haskell">.</span>0</span>)) <span class="keyword operator haskell">=</span> simplify e
simplify (<span class="constant other haskell">Add</span> e1 e2) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Add</span> (simplify e1) (simplify e2)
simplify (<span class="constant other haskell">Mul</span> e1 e2) <span class="keyword operator haskell">=</span> <span class="constant other haskell">Mul</span> (simplify e1) (simplify e2)
simplify e <span class="keyword operator haskell">=</span> e
</span></td></pre></tr></tbody></table></div>
<p>The second function will do all the work at the program's top level to compute the gradient and print it:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">showGradient</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Expr</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">IO</span> <span class="support constant unit haskell">()</span>
</span>showGradient func <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
putStrLn <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>f(..) = <span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> fshow func
forM_ (<span class="constant other haskell">Map</span><span class="keyword operator haskell">.</span>toList <span class="keyword operator haskell">$</span> gradient func) <span class="keyword operator haskell">$</span> <span class="keyword operator haskell">\</span>(k<span class="punctuation separator comma haskell">,</span> v) <span class="keyword operator haskell">-></span> <span class="keyword control haskell">do</span>
putStrLn <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>∂f / ∂<span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> fshow (<span class="constant other haskell">Term</span> k) <span class="keyword operator haskell">++</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span> = <span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> (fshow <span class="keyword operator haskell">.</span> simplify) v
</span></td></pre></tr></tbody></table></div><h2 id="does-it-work">Does it work?</h2>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="keyword operator haskell">></span> showGradient wikipediaFunc
f<span class="entity name function infix haskell">(..)</span> <span class="keyword operator haskell">=</span> (sin(x1) <span class="keyword operator haskell">+</span> (x1 * x2))
∂f <span class="keyword operator haskell">/</span> ∂x1 <span class="keyword operator haskell">=</span> (cos(x1) <span class="keyword operator haskell">+</span> x2)
∂f <span class="keyword operator haskell">/</span> ∂x2 <span class="keyword operator haskell">=</span> x1
</span></td></pre></tr></tbody></table></div>
<p>Looks that it does. We have arrived at the same results as Wikipedia.</p>
<script type="math/tex;mode=display">\begin{aligned}
& \frac{∂ f}{∂ x_1} = \cos x_1 + x_2 \\
& \frac{∂ f}{∂ x_2} = x_1 \\
\end{aligned}</script>
<p>Will it work with something more complex?</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="keyword operator haskell">></span> showGradient (<span class="constant other haskell">Sin</span> (<span class="constant other haskell">Mul</span> (<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">2</span> <span class="keyword operator function infix haskell"><span class="punctuation definition entity haskell">`</span>Add<span class="punctuation definition entity haskell">`</span></span> <span class="constant other haskell">Lit</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) <span class="keyword operator haskell">$</span> <span class="constant other haskell">Cos</span> (<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">1</span>))) <span class="keyword operator function infix haskell"><span class="punctuation definition entity haskell">`</span>Mul<span class="punctuation definition entity haskell">`</span></span> (<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">1</span>) <span class="keyword operator function infix haskell"><span class="punctuation definition entity haskell">`</span>Mul<span class="punctuation definition entity haskell">`</span></span> (<span class="constant other haskell">Term</span> <span class="constant numeric integer decimal haskell">3</span>)
f<span class="entity name function infix haskell">(..)</span> <span class="keyword operator haskell">=</span> ((sin(((x2 <span class="keyword operator haskell">+</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) * cos(x1))) * x1) * x3)
∂f <span class="keyword operator haskell">/</span> ∂x1 <span class="keyword operator haskell">=</span> (x3 * ((x1 * (cos(((x2 <span class="keyword operator haskell">+</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) * cos(x1))) * ((x2 <span class="keyword operator haskell">+</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) * <span class="keyword operator haskell">-</span>(sin(x1))))) <span class="keyword operator haskell">+</span> sin(((x2 <span class="keyword operator haskell">+</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) * cos(x1)))))
∂f <span class="keyword operator haskell">/</span> ∂x2 <span class="keyword operator haskell">=</span> (x3 * (x1 * (cos(((x2 <span class="keyword operator haskell">+</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) * cos(x1))) * cos(x1))))
∂f <span class="keyword operator haskell">/</span> ∂x3 <span class="keyword operator haskell">=</span> (sin(((x2 <span class="keyword operator haskell">+</span> <span class="constant numeric float decimal haskell">5<span class="punctuation separator decimal haskell">.</span>1</span>) * cos(x1))) * x1)
</span></td></pre></tr></tbody></table></div>
<p>Comparing with <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.wolframalpha.com/input/?i=derive+((sin(((x2+%2B+5.1)+*+cos(x1)))+*+x1)+*+x3)">Wolfram Alpha<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, it seems to get it right.</p>
<h2 id="end-note">End note</h2>
<p>Advance extensions of what I illustrated here can add a considerable amount of functionality and ease of use. We will definitely need to support matrices for instance, if we would like to derive a back-propagation graph. You can browse the <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://hackage.haskell.org/package/ad">ad<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> package to get some ideas.</p>
Intro2016-04-26T00:00:00+00:002016-04-26T00:00:00+00:00https://blog.aloni.org/pages/about/intro/<p>I'm a software developer, working as contractor for several companies. When I
first learned about the existence of computers I marveled at the ability to
program them, and so I taught myself the métier in the mid-1990's. I also
hold a B.Sc in computer science.</p>
<p>My expertise is a wide arch, ranging from low-level Linux kernel hacking and
machine code to compilers, Rust and Haskell programming, software packaging,
build automation, and more. I've been following Linux kernel development since
year 2000, and had the joy of adopting <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://git-scm.com/">Git<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> since its
inception in 2005.</p>
<p>I authored and contributed patches to various kinds of open source projects,
namely the Linux kernel (of which
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/Cooperative_Linux">coLinux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> was a project of mine
I presented in <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.kernel.org/doc/ols/2004/">OLS 2004<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>),
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://lists.gnu.org/archive/html/qemu-devel/2007-09/msg00010.html">QEMU<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>,
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/tmux/tmux/wiki">tmux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://neovim.io">Neovim<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>,
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/rust-lang/cargo">cargo<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>,
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/rust-lang/rust">rust<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, and many more. For more details, see
a comprehensive list of my <a href="/oss">OSS contributions</a>.</p>
<p>Other interests are playing and studying music, languages, and knowledge in
various fields of science.</p>
<p>My development environment is composed of
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Z_shell">zsh<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>,
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/tmux/tmux/wiki">tmux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> (with patches), with
<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://neovim.io">Neovim<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> as editor and <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/Fedora_%28operating_system%29">Fedora
Linux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> as a desktop
and laptop OS.</p>
Back Propagation with TensorFlow2016-04-26T00:00:00+00:002016-04-26T00:00:00+00:00https://blog.aloni.org/posts/backprop-with-tensorflow/<p><strong>(Updated for TensorFlow 1.0 on March 6th, 2017)</strong></p>
<p>When I first read about neural network in Michael Nielsen's <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://neuralnetworksanddeeplearning.com/">Neural Networks and Deep Learning<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, I was excited to find a good source that explains the material along with actual code. However there was a rather steep jump in the part that describes the basic math and the part that goes about implementing it, and it was especially apparent in the <code>numpy</code>-based code that implements backward propagation.</p>
<span id="continue-reading"></span>
<p>So, in order to explain it better to myself, and learn about <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.tensorflow.org/">TensorFlow<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> in the process, I took it upon myself to implement the first network in the book using TensorFlow by two means. First, <em>manually</em> defining the back propagation step, and the second - letting TensorFlow do the hard work using <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://en.wikipedia.org/wiki/Automatic_differentiation">automatic differentiation<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<h2 id="setup">Setup</h2>
<p>First, we need to load TensorFlow and setup the basic parts of the graph - inputs (<code>a_0</code>, <code>y</code>), and states (<code>w_1</code>, <code>b_1</code>, <code>w_2</code>, <code>b_2</code>).</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="comment line number-sign python"><span class="punctuation definition comment python">#</span>!/usr/bin/env python
</span>
<span class="meta statement import python"><span class="keyword control import python">import</span> <span class="meta qualified-name python"><span class="meta generic-name python">tensorflow</span></span></span>
<span class="meta statement import python"><span class="keyword control import python">import</span> <span class="meta qualified-name python"><span class="meta generic-name python">tensorflow</span></span> <span class="keyword control import as python">as</span> <span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span></span>
<span class="meta statement import python"><span class="keyword control import from python">from</span></span><span class="meta statement import python"><span class="meta import-source python"> <span class="meta import-path python"><span class="meta import-name python">tensorflow</span><span class="punctuation accessor dot python">.</span><span class="meta import-name python">examples</span><span class="punctuation accessor dot python">.</span><span class="meta import-name python">tutorials</span><span class="punctuation accessor dot python">.</span><span class="meta import-name python">mnist</span></span> <span class="meta statement import python"><span class="keyword control import python">import</span></span></span></span><span class="meta statement import python"></span><span class="meta statement import python"> <span class="meta generic-name python">input_data</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">mnist</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">input_data</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">read_data_sets</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta string python"><span class="string quoted double python"><span class="punctuation definition string begin python">"</span></span></span><span class="meta string python"><span class="string quoted double python">MNIST_data/<span class="punctuation definition string end python">"</span></span></span><span class="punctuation separator arguments python">,</span> <span class="variable parameter python">one_hot</span><span class="keyword operator assignment python">=</span><span class="constant language python">True</span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">a_0</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">placeholder</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">tf</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">float32</span></span><span class="punctuation separator arguments python">,</span> <span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant language python">None</span><span class="punctuation separator sequence python">,</span> <span class="constant numeric integer decimal python">784</span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">y</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">placeholder</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">tf</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">float32</span></span><span class="punctuation separator arguments python">,</span> <span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant language python">None</span><span class="punctuation separator sequence python">,</span> <span class="constant numeric integer decimal python">10</span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">middle</span></span> <span class="keyword operator assignment python">=</span> <span class="constant numeric integer decimal python">30</span>
<span class="meta qualified-name python"><span class="meta generic-name python">w_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">Variable</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">truncated_normal</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant numeric integer decimal python">784</span><span class="punctuation separator sequence python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">middle</span></span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">b_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">Variable</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">truncated_normal</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant numeric integer decimal python">1</span><span class="punctuation separator sequence python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">middle</span></span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">w_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">Variable</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">truncated_normal</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="meta qualified-name python"><span class="meta generic-name python">middle</span></span><span class="punctuation separator sequence python">,</span> <span class="constant numeric integer decimal python">10</span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">b_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">Variable</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">truncated_normal</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant numeric integer decimal python">1</span><span class="punctuation separator sequence python">,</span> <span class="constant numeric integer decimal python">10</span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="the-sigmoid-function">The sigmoid function</h2>
<p>Our sigmoid function, although provided by TensorFlow's extensive function library, is brought here as reference:</p>
<p>$$
\sigma(z) = \frac{1}{1+e^{-z}}
$$</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta function python"><span class="storage type function python"><span class="keyword declaration function python">def</span></span> <span class="entity name function python"><span class="meta generic-name python">sigma</span></span></span><span class="meta function parameters python"><span class="punctuation section parameters begin python">(</span></span><span class="meta function parameters python"><span class="variable parameter python">x</span><span class="punctuation section parameters end python">)</span></span><span class="meta function python"><span class="punctuation section function begin python">:</span></span>
<span class="keyword control flow return python">return</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">div</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">constant</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric float decimal python">1<span class="punctuation separator decimal python">.</span>0</span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">add</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">constant</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric float decimal python">1<span class="punctuation separator decimal python">.</span>0</span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">exp</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">negative</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">x</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="the-forward-propagation">The forward propagation</h2>
<p>Provided that the input image is given by the \( a_0 \) matrix, calculating forward propagation for multiple
images at a time can be done with simple matrix multiplication, defined as such:</p>
<script type="math/tex;mode=display">\begin{aligned}
& z_1 = a_0 \cdot w_1 + b_1 \\
& a_1 = \sigma(z_1) \\
& z_2 = a_1 \cdot w_2 + b_2 \\
& a_2 = \sigma(z_2) \\
\end{aligned}</script>
<p>Given a tensor of <em>multiple images</em>, this can done in TensorFlow for <em>all</em> them at the same time (thanks to 'broadcasting'),
so the above gets a one-to-one translation in TensorFlow:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta qualified-name python"><span class="meta generic-name python">z_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">add</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">matmul</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">a_0</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">w_1</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">b_1</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">a_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">sigma</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">z_1</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">z_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">add</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">matmul</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">a_1</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">w_2</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">b_2</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">a_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">sigma</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">z_2</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="difference">Difference</h2>
<p>The input provides \( y \) as the test for the accuracy of the network's output, so we compute the following vector:</p>
<script type="math/tex;mode=display">\begin{aligned}
& \nabla a = a_2 - y \\
\end{aligned}</script>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta qualified-name python"><span class="meta generic-name python">diff</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">subtract</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">a_2</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">y</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="the-sigmoid-prime-function">The sigmoid prime function</h2>
<p>Here's the derivate of the sigmoid function from above, which will be needed during the backward propagation:</p>
<p>$$
\sigma'(z) = \sigma(z)(1 - \sigma(z))
$$</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta function python"><span class="storage type function python"><span class="keyword declaration function python">def</span></span> <span class="entity name function python"><span class="meta generic-name python">sigmaprime</span></span></span><span class="meta function parameters python"><span class="punctuation section parameters begin python">(</span></span><span class="meta function parameters python"><span class="variable parameter python">x</span><span class="punctuation section parameters end python">)</span></span><span class="meta function python"><span class="punctuation section function begin python">:</span></span>
<span class="keyword control flow return python">return</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">sigma</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">x</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">subtract</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">constant</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric float decimal python">1<span class="punctuation separator decimal python">.</span>0</span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">sigma</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">x</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="back">Backward propagation</h2>
<p>The most complicated part is the backward propagation. First, we need to compute the deltas of the weights and biases. In the original book the Python code was a bit puzzling, but here we can describe the same algorithm in a functional, stateless way.</p>
<script type="math/tex;mode=display">\begin{aligned}
& \nabla z_2 = \nabla a \cdot \sigma'(z_2) \\
& \nabla b_2 = \nabla z_2 \\
& \nabla w_2 = a_1^T \cdot \nabla z_2 \\
& \\
& \nabla a_1 = \nabla z_2 \cdot w_2^T \\
& \nabla z_1 = \nabla a_1 \cdot \sigma'(z_1) \\
& \nabla b_1 = \nabla z_1 \\
& \nabla w_1 = a_0^T \cdot \nabla z_1 \\
\end{aligned}</script>
<p>It's also one-to-one with:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta qualified-name python"><span class="meta generic-name python">d_z_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">diff</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">sigmaprime</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">z_2</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">d_b_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta qualified-name python"><span class="meta generic-name python">d_z_2</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">d_w_2</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">matmul</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">transpose</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">a_1</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">d_z_2</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">d_a_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">matmul</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">d_z_2</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">transpose</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">w_2</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">d_z_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">d_a_1</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">sigmaprime</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">z_1</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">d_b_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta qualified-name python"><span class="meta generic-name python">d_z_1</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">d_w_1</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">matmul</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">transpose</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">a_0</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">d_z_1</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="upd">Updating the network</h2>
<p>We take the computed \( \nabla \) and update the weights in one step. Note that the following does not precisely match the book - I have omitted the constant \( 1/n \) divider. For simplicity, it's not really needed, as it comes into play inside the \( \eta \) itself, in this case.</p>
<script type="math/tex;mode=display">\begin{aligned}
& w_1 \leftarrow w_1 - \eta \cdot \nabla w_1 \\
& b_1 \leftarrow b_1 - \eta \cdot \nabla b_1 \\
& w_2 \leftarrow w_2 - \eta \cdot \nabla w_2 \\
& b_2 \leftarrow b_2 - \eta \cdot \nabla b_2 \\
\end{aligned}</script>
<p>In TensorFlow, it can translate to a list of a assignments:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta qualified-name python"><span class="meta generic-name python">eta</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">constant</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric float decimal python">0<span class="punctuation separator decimal python">.</span>5</span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">step</span></span> <span class="keyword operator assignment python">=</span> <span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">assign</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">w_1</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">subtract</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">w_1</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">eta</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">d_w_1</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="punctuation separator sequence python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">assign</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">b_1</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">subtract</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">b_1</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">eta</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">reduce_mean</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">d_b_1</span></span><span class="punctuation separator arguments python">,</span> <span class="variable parameter python">axis</span><span class="keyword operator assignment python">=</span><span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant numeric integer decimal python">0</span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="punctuation separator sequence python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">assign</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">w_2</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">subtract</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">w_2</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">eta</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">d_w_2</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="punctuation separator sequence python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">assign</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">b_2</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">subtract</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">b_2</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">eta</span></span><span class="punctuation separator arguments python">,</span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">reduce_mean</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">d_b_2</span></span><span class="punctuation separator arguments python">,</span> <span class="variable parameter python">axis</span><span class="keyword operator assignment python">=</span><span class="meta sequence list python"><span class="punctuation section sequence begin python">[</span><span class="constant numeric integer decimal python">0</span><span class="punctuation section sequence end python">]</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="punctuation section sequence end python">]</span></span>
</span></td></pre></tr></tbody></table></div><h2 id="running-and-testing-the-training-process">Running and testing the training process</h2>
<p>The following will be able to train the network and test it in the meanwhile, using mini-batches of 10. Here, I chose to test with 1000 items from the test set, every 1000 mini-batches.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta qualified-name python"><span class="meta generic-name python">acct_mat</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">equal</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">argmax</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">a_2</span></span><span class="punctuation separator arguments python">,</span> <span class="constant numeric integer decimal python">1</span><span class="punctuation section arguments end python">)</span></span><span class="punctuation separator arguments python">,</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">argmax</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">y</span></span><span class="punctuation separator arguments python">,</span> <span class="constant numeric integer decimal python">1</span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">acct_res</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">reduce_sum</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">cast</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">acct_mat</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">tf</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">float32</span></span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">sess</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">InteractiveSession</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="punctuation section arguments end python">)</span></span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">sess</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">run</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">global_variables_initializer</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="punctuation section arguments end python">)</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta statement loop for python"><span class="keyword control loop for python">for</span> <span class="meta generic-name python">i</span> <span class="keyword control loop for in python">in</span></span><span class="meta statement loop for python"> <span class="meta function-call python"><span class="meta qualified-name python"><span class="variable function python"><span class="support type python">xrange</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric integer decimal python">10000</span><span class="punctuation section arguments end python">)</span></span></span><span class="meta statement loop for python"><span class="punctuation section block loop for python">:</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">batch_xs</span></span>, <span class="meta qualified-name python"><span class="meta generic-name python">batch_ys</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">mnist</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">train</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">next_batch</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric integer decimal python">10</span><span class="punctuation section arguments end python">)</span></span>
<span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">sess</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">run</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">step</span></span><span class="punctuation separator arguments python">,</span> <span class="variable parameter python">feed_dict</span> <span class="keyword operator assignment python">=</span> <span class="meta mapping python"><span class="punctuation section mapping begin python">{</span></span><span class="meta mapping key python"><span class="meta qualified-name python"><span class="meta generic-name python">a_0</span></span></span><span class="meta mapping python"><span class="punctuation separator mapping key-value python">:</span></span><span class="meta mapping python"> </span><span class="meta mapping value python"><span class="meta qualified-name python"><span class="meta generic-name python">batch_xs</span></span></span><span class="meta mapping python"><span class="punctuation separator mapping python">,</span>
<span class="meta mapping key python"><span class="meta qualified-name python"><span class="meta generic-name python">y</span></span> </span><span class="punctuation separator mapping key-value python">:</span></span><span class="meta mapping python"> </span><span class="meta mapping value python"><span class="meta qualified-name python"><span class="meta generic-name python">batch_ys</span></span></span><span class="meta mapping python"><span class="punctuation section mapping end python">}</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta statement conditional if python"><span class="keyword control conditional if python">if</span> <span class="meta qualified-name python"><span class="meta generic-name python">i</span></span> <span class="keyword operator arithmetic python">%</span> <span class="constant numeric integer decimal python">1000</span> <span class="keyword operator comparison python">==</span> <span class="constant numeric integer decimal python">0</span><span class="punctuation section block conditional if python">:</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">res</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">sess</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">run</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">acct_res</span></span><span class="punctuation separator arguments python">,</span> <span class="variable parameter python">feed_dict</span> <span class="keyword operator assignment python">=</span>
<span class="meta mapping python"><span class="punctuation section mapping begin python">{</span></span><span class="meta mapping key python"><span class="meta qualified-name python"><span class="meta generic-name python">a_0</span></span></span><span class="meta mapping python"><span class="punctuation separator mapping key-value python">:</span></span><span class="meta mapping python"> </span><span class="meta mapping value python"><span class="meta item-access python"><span class="meta qualified-name python"><span class="meta generic-name python">mnist</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">test</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">images</span></span></span><span class="meta item-access python"><span class="punctuation section brackets begin python">[</span></span><span class="meta item-access arguments python"><span class="punctuation separator slice python">:</span><span class="constant numeric integer decimal python">1000</span></span><span class="meta item-access python"><span class="punctuation section brackets end python">]</span></span></span><span class="meta mapping python"><span class="punctuation separator mapping python">,</span>
<span class="meta mapping key python"><span class="meta qualified-name python"><span class="meta generic-name python">y</span></span> </span><span class="punctuation separator mapping key-value python">:</span></span><span class="meta mapping python"> </span><span class="meta mapping value python"><span class="meta item-access python"><span class="meta qualified-name python"><span class="meta generic-name python">mnist</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">test</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">labels</span></span></span><span class="meta item-access python"><span class="punctuation section brackets begin python">[</span></span><span class="meta item-access arguments python"><span class="punctuation separator slice python">:</span><span class="constant numeric integer decimal python">1000</span></span><span class="meta item-access python"><span class="punctuation section brackets end python">]</span></span></span><span class="meta mapping python"><span class="punctuation section mapping end python">}</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="keyword other print python">print</span> <span class="meta qualified-name python"><span class="meta generic-name python">res</span></span>
</span></td></pre></tr></tbody></table></div>
<p>Running it shows that it manages to train the network, as we quickly get 923 correct out of 1000 tests.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
93.0
741.0
870.0
909.0
904.0
912.0
916.0
923.0
</span></td></pre></tr></tbody></table></div><h2 id="automatic-differentiation">Automatic differentiation</h2>
<p>The great part about TensorFlow is its ability to derive the step function on its own. So, instead of the rather complicated 'Backward propagation' and 'Updating the network' given above for educational purposes (see: <a href="https://blog.aloni.org/posts/backprop-with-tensorflow/#back">backward propagation</a> and <a href="https://blog.aloni.org/posts/backprop-with-tensorflow/#upd">updating</a>), we can simply write the following step function alternative:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source python"><span class="meta qualified-name python"><span class="meta generic-name python">cost</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">multiply</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">diff</span></span><span class="punctuation separator arguments python">,</span> <span class="meta qualified-name python"><span class="meta generic-name python">diff</span></span><span class="punctuation section arguments end python">)</span></span>
<span class="meta qualified-name python"><span class="meta generic-name python">step</span></span> <span class="keyword operator assignment python">=</span> <span class="meta function-call python"><span class="meta qualified-name python"><span class="meta generic-name python">tf</span><span class="punctuation accessor dot python">.</span><span class="meta generic-name python">train</span></span><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">GradientDescentOptimizer</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="constant numeric float decimal python">0<span class="punctuation separator decimal python">.</span>1</span><span class="punctuation section arguments end python">)</span></span><span class="meta function-call python"><span class="meta qualified-name python"><span class="punctuation accessor dot python">.</span></span><span class="meta qualified-name python"><span class="variable function python"><span class="meta generic-name python">minimize</span></span></span></span><span class="meta function-call arguments python"><span class="punctuation section arguments begin python">(</span><span class="meta qualified-name python"><span class="meta generic-name python">cost</span></span><span class="punctuation section arguments end python">)</span></span>
</span></td></pre></tr></tbody></table></div>
<p>And observe that the training still works.</p>
Mailing Lists Under GMail2016-03-15T00:00:00+00:002016-03-15T00:00:00+00:00https://blog.aloni.org/posts/mailing-lists-under-gmail/<p>In the Webmail era, E-Mail for group correspondences is often mishandled. Here's how to efficiently handle it in GMail.</p>
<span id="continue-reading"></span>
<p>With many open source projects, you may innocuously subscribe to an Mailing list on the Internet, or you may be unwittingly added to a group mail in an organization (those are often the <code>all@</code> and <code>dev@</code> and <code>rnd@</code> E-Mail prefixes). Your in-box is now bombed with messages that are not for you, in discussions you don't (yet) participate, but some of them are or would be, later.</p>
<p>At that point, you should be made aware of the situation and use GMail filters because they are easy to configure. However, the filters you choose play a crucial part on how the mailing list is handled. What so are the options?</p>
<p><strong>The bad options</strong></p>
<ul>
<li><strong>Bad</strong> option #1: Skip in-box for everything sent to the mailing list. Now your GMail archive is some unsorted junk.</li>
<li><strong>Bad</strong> option #2: Delete everything sent to the mailing list that is not for me. Now, you cannot interject on discussions that could be relevant to your work.</li>
<li><strong>Bad</strong> option #3: Tag all posts to the mailing list with a label, and skip the in-box. We are getting there - however it's still bad. Because messages to you are archived as well (i.e. disappeared from in-box but still searchable, in GMail's terminology).</li>
</ul>
<p><strong>One good option</strong></p>
<p>What we should expect from GMail is the following:</p>
<ul>
<li>If a group E-Mail is received, then:
<ul>
<li>If your personal E-Mail address is <em>not also</em> in the <code>To:</code> or <code>Cc:</code> fields, then <em>skip in-box</em> and <em>apply a label</em>. The GMail will remain is unread for later consumption.</li>
<li>Otherwise, <em>only apply the label</em> but <em>don't skip the in-box</em>. Since you are personally addressed in addition to the group, the discussion is relevant and your response is needed.</li>
</ul>
</li>
</ul>
<p>The nice thing about Labels in GMail is that they map nicely to groups under this scheme. When you visit the label, you are essentially visiting the group, and you can join discussions. One of nice benefits is that GMail tracks which discussions were read or unread by you, without cluttering your inbox.</p>
<p><strong>Adding GMail filters for Mailing List that Just Do The Proper Job:</strong></p>
<p>When viewing details for an E-Mail, GMail shows a link 'Filter messages from this mailing list'. It presents you with a bad filter that you can edit. However I think it's better to add the good set of filters from the outset instead of editing bad ones.</p>
<p>Go to one of the group E-Mails, and do <em>Show Original</em>, then look for the <code>List-Id</code> header. It would most likely appear similarly to this:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">List-ID: <group.company.com>
</span></td></pre></tr></tbody></table></div>
<p>To go along the rules of the option above, we should have filters that look like below, in the GMail's filter settings:</p>
<img src="/images/gmail-filter-settings.png" class="center">
<p>We can manually enter these filters in a few steps.</p>
<ul>
<li>
<p>The first GMail filter is (after relevant <strong>strings replaced</strong>):</p>
<p><code>list:group.company.com to:your.name@organization.com</code></p>
</li>
</ul>
<p><strong>How to enable:</strong> Enter in GMail search box, hit tiny arrow in the right of the search box -> <em>Show Search Options</em> -> <em>Create As Filter</em> -> Do this: Apply label "Group Name"</p>
<ul>
<li>
<p>The second GMail filter is (after relevant <strong>strings replaced</strong>):</p>
<p><code>list:group.company.com -to:your.name@organization.com</code></p>
</li>
</ul>
<p><strong>How to enable:</strong> Enter in GMail search box, hit tiny arrow in the right of the search box -> <em>Show Search Options</em> -> <em>Create As Filter</em> -> Do this: Skip In-box, Apply label "Group Name"</p>
<p>Alternatively, instead of the manual filter entry, it's possible to edit the following XML, and import it under GMail's filter settings.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text xml"><span class="meta tag preprocessor xml"><span class="punctuation definition tag begin xml"><?</span><span class="entity name tag xml">xml</span> <span class="entity other attribute-name localname xml">version</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>1.0<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">encoding</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>UTF-8<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">?></span></span><span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">feed</span> <span class="entity other attribute-name localname xml">xmlns</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>http://www.w3.org/2005/Atom<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name namespace xml">xmlns</span><span class="entity other attribute-name xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity other attribute-name localname xml">apps</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>http://schemas.google.com/apps/2006<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">title</span><span class="punctuation definition tag end xml">></span></span>Mail Filters<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">title</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">id</span><span class="punctuation definition tag end xml">></span></span>tag:mail.google.com,2008:filters:1458024047214,1458024071178<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">id</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">entry</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">category</span> <span class="entity other attribute-name localname xml">term</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>filter<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">></span></span><span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">category</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">title</span><span class="punctuation definition tag end xml">></span></span>Mail Filter<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">title</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">id</span><span class="punctuation definition tag end xml">></span></span>tag:mail.google.com,2008:filter:1458024047214<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">id</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>hasTheWord<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>list:group.company.com to:your.name@organization.com<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>label<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>Dev<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>sizeOperator<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>s_sl<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>sizeUnit<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>s_smb<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">entry</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">entry</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">category</span> <span class="entity other attribute-name localname xml">term</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>filter<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">></span></span><span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">category</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">title</span><span class="punctuation definition tag end xml">></span></span>Mail Filter<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">title</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag localname xml">id</span><span class="punctuation definition tag end xml">></span></span>tag:mail.google.com,2008:filter:1458024071178<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">id</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>hasTheWord<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>list:group.company.com -to:your.name@organization.com<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>label<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>Dev<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>shouldArchive<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>true<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>sizeOperator<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>s_sl<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"><</span><span class="entity name tag namespace xml">apps</span><span class="entity name tag xml"><span class="punctuation separator namespace xml">:</span></span><span class="entity name tag localname xml">property</span> <span class="entity other attribute-name localname xml">name</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>sizeUnit<span class="punctuation definition string end xml">'</span></span> <span class="entity other attribute-name localname xml">value</span><span class="punctuation separator key-value xml">=</span><span class="string quoted single xml"><span class="punctuation definition string begin xml">'</span>s_smb<span class="punctuation definition string end xml">'</span></span><span class="punctuation definition tag end xml">/></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">entry</span><span class="punctuation definition tag end xml">></span></span>
<span class="meta tag xml"><span class="punctuation definition tag begin xml"></</span><span class="entity name tag localname xml">feed</span><span class="punctuation definition tag end xml">></span></span>
</span></td></pre></tr></tbody></table></div>Performance of the ST Monad with Pure Exceptions2016-02-20T00:00:00+00:002016-02-20T00:00:00+00:00https://blog.aloni.org/posts/st-monad-perf-with-exceptions/<p>The <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://wiki.haskell.org/Monad/ST">ST Monad<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> provides a venerable method in Haskell for writing stateful imperative code. Writing such code, in contrast to the non-stateful approach, is sometimes better. Some algorithms are better understood or better illustrated with states, and another reason is increased performance. The difference between <code>ST</code> and <code>IO</code> is important, because when we implement an algorithm, we only want to deal with the internal states and not bother with side effects that don't belong to it. Allowing stateful algorithm to remain pure under <code>ST</code>, gives way to better code generation by the compiler.</p>
<span id="continue-reading"></span>
<p>Some algorithms are better written with <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://wiki.haskell.org/Exception">Exceptions<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>. For example, an algorithm for validating an expression tree may be such one. However, one needs to be aware that exceptions in pure code can only be caught in <code>IO</code>, unless <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://hackage.haskell.org/package/exceptions">pure exceptions<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> are used. We can use these pure exceptions under a <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://book.realworldhaskell.org/read/monad-transformers.html">monad transformer<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, but then we need to verify that there was no significant loss in performance. We suspect that the <code>CatchT</code> transformer would provide us a zero-cost abstraction, being a <code>newtype</code>. But how well would GHC succeed in optimizing away the transformations?</p>
<h2 id="fibonacci-that-throws-for-kicks">Fibonacci that throws, for kicks</h2>
<p>First, let us look at the example for <code>ST</code> brought from the <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://wiki.haskell.org/Monad/ST">Haskell Wiki<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, which presents us with the stateful implementation of computing the n-th Fibonacci number:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">fibST</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Integer</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">Integer</span>
</span>fibST n <span class="keyword operator haskell">=</span>
<span class="keyword control haskell">if</span> n <span class="keyword operator haskell"><</span> <span class="constant numeric integer decimal haskell">2</span>
<span class="keyword control haskell">then</span> n
<span class="keyword control haskell">else</span> runST <span class="keyword operator haskell">$</span> <span class="keyword control haskell">do</span> x <span class="keyword operator haskell"><-</span> newSTRef <span class="constant numeric integer decimal haskell">0</span>
y <span class="keyword operator haskell"><-</span> newSTRef <span class="constant numeric integer decimal haskell">1</span>
fibST' n x y
<span class="keyword other haskell">where</span> fibST' <span class="constant numeric integer decimal haskell">0</span> x _ <span class="keyword operator haskell">=</span> readSTRef x
fibST' n' x y <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
x' <span class="keyword operator haskell"><-</span> readSTRef x
y' <span class="keyword operator haskell"><-</span> readSTRef y
writeSTRef x y'
writeSTRef y <span class="keyword operator haskell">$!</span> x'<span class="keyword operator haskell">+</span>y'
fibST' (n' <span class="keyword operator haskell">-</span> <span class="constant numeric integer decimal haskell">1</span>) x y
</span></td></pre></tr></tbody></table></div>
<p>We would like to test the performance of pure exceptions. So, let us have a slightly modified version of it, being a modulo of Fibonacci using <code>Int</code>. The change of type from <code>Integer</code> to <code>Int</code> would be better for us when measuring performance, otherwise the run would have spent time adding really big numbers in each one of the loop iterations.</p>
<p>We will use this new <code>Exception</code> type,</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="keyword other haskell">data</span> <span class="constant other haskell">MyException</span> <span class="keyword operator haskell">=</span> <span class="constant other haskell">MyException</span> <span class="constant other haskell">Int</span>
<span class="keyword other haskell">deriving</span> <span class="constant other haskell">Show</span>
<span class="meta declaration instance haskell"><span class="keyword other haskell">instance</span> <span class="storage type haskell">Exception</span> <span class="storage type haskell">MyException</span></span>
</span></td></pre></tr></tbody></table></div>
<p>... and modify the function in various ways:</p>
<ul>
<li>Have the caller use <code>runST</code>.</li>
<li>Change the type signature bear <code>Int</code> and be under <code>ST</code>.</li>
<li>Add a throw to some exception along the way.</li>
</ul>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="sourceMarker">fibMod <span class="keyword operator haskell">::</span> <span class="constant other haskell">Int</span> <span class="keyword operator haskell">-></span> <span class="constant other haskell">ST</span> s <span class="constant other haskell">Int</span></span>
<span class="sourceMarker">fibMod n <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span></span>
<span class="keyword control haskell">if</span> n <span class="keyword operator haskell"><</span> <span class="constant numeric integer decimal haskell">2</span>
<span class="keyword control haskell">then</span> <span class="sourceMarker">return n</span>
<span class="keyword control haskell">else</span> <span class="keyword control haskell">do</span> x <span class="keyword operator haskell"><-</span> newSTRef <span class="constant numeric integer decimal haskell">0</span>
y <span class="keyword operator haskell"><-</span> newSTRef <span class="constant numeric integer decimal haskell">1</span>
fibMod' n x y
<span class="keyword other haskell">where</span>
fibMod' <span class="constant numeric integer decimal haskell">0</span> x _ <span class="keyword operator haskell">=</span> readSTRef x
fibMod' n' x y <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
x' <span class="keyword operator haskell"><-</span> readSTRef x
y' <span class="keyword operator haskell"><-</span> readSTRef y
<span class="sourceMarker">when (n' <span class="keyword operator haskell">==</span> <span class="constant numeric integer decimal haskell">1000</span>) <span class="keyword operator haskell">$</span> <span class="keyword control haskell">do</span></span>
<span class="sourceMarker">throw <span class="keyword operator haskell">$</span> <span class="constant other haskell">MyException</span> x' <span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> not a pure exception (yet!)</span>
</span> writeSTRef x y'
writeSTRef y <span class="keyword operator haskell">$!</span> x'<span class="keyword operator haskell">+</span>y'
fibMod' (n'<span class="keyword operator haskell">-</span><span class="constant numeric integer decimal haskell">1</span>) x y
</span></td></pre></tr></tbody></table></div>
<p>We have yet to add a <code>catch</code> anywhere. The problem is that we cannot have a pure function in <code>ST</code> doing <code>catch</code>, because <code>catch :: (Exception e, MonadCatch m) => m a -> (e -> m a) -> m a</code>. If we try to use <code>catch</code>, we will get the error <code>No instance for (MonadCatch (ST s))</code>.</p>
<p>To solve, we shall use the <code>ST</code> monad with <code>CatchT</code>. First, we define <code>STCatch</code> type synonym for a short hand:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="keyword other haskell">type</span> <span class="constant other haskell">STCatch</span> s a <span class="keyword operator haskell">=</span> <span class="constant other haskell">CatchT</span> (<span class="constant other haskell">ST</span> s) a
</span></td></pre></tr></tbody></table></div>
<p>Now, let us create our <code>fibMod_E</code> variant, which is under <code>STCatch</code>, and modify it to use pure exceptions, thrown using <code>throwM</code>. We will also add a <code>catch</code> wrap, which will fix <code>fibMod_E</code> to return <code>-1</code> on the thrown exception. The <code>catch</code> is conditional, so we can see its effect depending on the use.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">fibMod_E</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Int</span> <span class="keyword other arrow haskell">-></span> <span class="sourceMarker"><span class="storage type haskell">STCatch</span></span> <span class="variable other generic-type haskell">s</span> <span class="storage type haskell">Int</span>
</span>fibMod_E n <span class="keyword operator haskell">=</span>
<span class="keyword control haskell">if</span> n <span class="keyword operator haskell"><</span> <span class="constant numeric integer decimal haskell">2</span>
<span class="keyword control haskell">then</span> return n
<span class="keyword control haskell">else</span> <span class="keyword control haskell">do</span> x <span class="keyword operator haskell"><-</span> <span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> newSTRef <span class="constant numeric integer decimal haskell">0</span>
y <span class="keyword operator haskell"><-</span> <span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> newSTRef <span class="constant numeric integer decimal haskell">1</span>
fibMod' n x y
<span class="keyword other haskell">where</span>
fibMod' <span class="constant numeric integer decimal haskell">0</span> x _ <span class="keyword operator haskell">=</span> <span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> readSTRef x
fibMod' n' x y <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
x' <span class="keyword operator haskell"><-</span> <span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> readSTRef x
y' <span class="keyword operator haskell"><-</span> <span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> readSTRef y
<span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> writeSTRef x y'
<span class="sourceMarker">lift <span class="keyword operator haskell">$</span></span> writeSTRef y <span class="keyword operator haskell">$!</span> x'<span class="keyword operator haskell">+</span>y'
when (abs n' <span class="keyword operator haskell">==</span> <span class="constant numeric integer decimal haskell">1000</span>) <span class="keyword operator haskell">$</span> <span class="keyword control haskell">do</span>
<span class="sourceMarker">throwM</span> <span class="keyword operator haskell">$</span> <span class="constant other haskell">MyException</span> x'
<span class="sourceMarker"><span class="keyword other haskell">let</span> recurse <span class="keyword operator haskell">=</span></span> fibMod' (n'<span class="keyword operator haskell">-</span><span class="constant numeric integer decimal haskell">1</span>) x y
<span class="sourceMarker"><span class="keyword control haskell">if</span> n <span class="keyword operator haskell"><=</span> <span class="constant numeric integer decimal haskell">25000000</span></span>
<span class="sourceMarker"><span class="keyword control haskell">then</span> recurse</span>
<span class="sourceMarker"><span class="keyword control haskell">else</span> catch recurse (<span class="keyword operator haskell">\</span>(<span class="constant other haskell">MyException</span> _) <span class="keyword operator haskell">-></span> return (<span class="keyword operator haskell">-</span><span class="constant numeric integer decimal haskell">1</span>))</span>
</span></td></pre></tr></tbody></table></div>
<p>Can we degrade an <code>ST</code> exception back to an <code>IO</code> exception? Yes! Using the following function, that requires the <code>RankNTypes</code> extension for its type signature:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> | A variant of `runST` for STCatch that turns all _uncaught_
</span><span class="comment line double-dash haskell"><span class="punctuation definition comment haskell">--</span> 'throwM' exceptions back to exceptions thrown in `IO`.
</span><span class="meta function type-declaration haskell"><span class="entity name function haskell">runSTthrowIO</span> <span class="keyword other double-colon haskell">::</span> (<span class="variable other generic-type haskell">forall</span> <span class="variable other generic-type haskell">s</span>. <span class="storage type haskell">STCatch</span> <span class="variable other generic-type haskell">s</span> <span class="variable other generic-type haskell">a</span>) <span class="keyword other arrow haskell">-></span> <span class="variable other generic-type haskell">a</span>
</span>runSTthrowIO action <span class="keyword operator haskell">=</span>
<span class="keyword other haskell">case</span> runST <span class="keyword operator haskell">$</span> runCatchT action <span class="keyword other haskell">of</span>
<span class="constant other haskell">Left</span> e <span class="keyword operator haskell">-></span> throw e
<span class="constant other haskell">Right</span> r <span class="keyword operator haskell">-></span> r
</span></td></pre></tr></tbody></table></div><h2 id="testing">Testing</h2>
<p>Now we are ready for testing. We will use the following utility, depending on <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://hackage.haskell.org/package/criterion">criterion<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">timeIt</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">IO</span> <span class="support constant unit haskell">()</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">IO</span> <span class="support constant unit haskell">()</span>
</span>timeIt act <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
<span class="keyword other haskell">let</span> w'act <span class="keyword operator haskell">=</span> whnfIO <span class="keyword operator haskell">$</span> catch act err
err <span class="keyword operator haskell">=</span> (<span class="keyword operator haskell">\</span>e@(<span class="constant other haskell">MyException</span> _) <span class="keyword operator haskell">-></span> putStrLn <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>caught: <span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> show e)
t <span class="keyword operator haskell"><-</span> measure w'act <span class="constant numeric integer decimal haskell">1</span>
putStrLn <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Total time: <span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> show (measTime <span class="keyword operator haskell">$</span> fst t)
</span></td></pre></tr></tbody></table></div>
<p>We shall test with various recursion depths, on the two fuctions under discussion:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">main</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">IO</span> <span class="support constant unit haskell">()</span>
</span>main <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>------ With lots of catches<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runST <span class="keyword operator haskell">$</span> fibMod <span class="constant numeric integer decimal haskell">50000000</span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runSTthrowIO <span class="keyword operator haskell">$</span> fibMod_E <span class="constant numeric integer decimal haskell">50000000</span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="constant character escape haskell">\n</span>------ With just one catch<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runST <span class="keyword operator haskell">$</span> fibMod <span class="constant numeric integer decimal haskell">25000001</span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runSTthrowIO <span class="keyword operator haskell">$</span> fibMod_E <span class="constant numeric integer decimal haskell">25000001</span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="constant character escape haskell">\n</span>------ With no catch<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runST <span class="keyword operator haskell">$</span> fibMod <span class="constant numeric integer decimal haskell">25000000</span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runSTthrowIO <span class="keyword operator haskell">$</span> fibMod_E <span class="constant numeric integer decimal haskell">25000000</span>
</span></td></pre></tr></tbody></table></div>
<p>And the result is:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">------ With lots of catches
caught: MyException (-5541175486947481557)
Total time: 0.6768422469031066
-1
Total time: 1.5523557260166854
------ With just one catch
caught: MyException 5934185968946882193
Total time: 0.32111207104753703
-1
Total time: 0.7558517289580777
------ With no catch
caught: MyException 4809429493926266912
Total time: 0.32045713800471276
caught: MyException 4809429493926266912
<span class="sourceMarker">Total time: 0.26791215199045837</span>
</span></td></pre></tr></tbody></table></div>
<p>The first two results are of no surprise. Both the <code>if</code> and <code>catch</code> incur their overheads. The last result is more peculiar, because it suggests that the code for <code>fibMod_E</code> emanated from the compiler is <em>even faster</em>, despite of the <code>if</code>, as long as there are no wrapping <code>catch</code>'s in the evaluation. The difference probably boils down to the generated machine code, but I'd leave that to a topic of a different post.</p>
<h2 id="a-few-extra-tests">A few extra tests</h2>
<p>(edit: added March 2, 2016)</p>
<p>Edward Kmett <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.reddit.com/r/haskell/comments/47evoh/performance_of_the_st_monad_with_pure_exceptions/d0cj9z1">pointed out<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> on Reddit that perhaps it would be interesting to test with <code>unsafeSTToIO</code>. So I've added the following cases (and also regenerated the results above, because every little change can affect the optimizer, and they varied slightly).</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"> putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>------ With lots of catches<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> (unsafeSTToIO <span class="keyword operator haskell">$</span> fibMod <span class="constant numeric integer decimal haskell">50000000</span>) <span class="keyword operator haskell">>>=</span> print
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="constant character escape haskell">\n</span>------ With just one catch<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> (unsafeSTToIO <span class="keyword operator haskell">$</span> fibMod <span class="constant numeric integer decimal haskell">25000001</span>) <span class="keyword operator haskell">>>=</span> print
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="constant character escape haskell">\n</span>------ With no catch<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> (unsafeSTToIO <span class="keyword operator haskell">$</span> fibMod <span class="constant numeric integer decimal haskell">25000000</span>) <span class="keyword operator haskell">>>=</span> print
</span></td></pre></tr></tbody></table></div>
<p>The additional output is:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">------ With lots of catches
caught: MyException (-5541175486947481557)
Total time: 0.5766234899638221
------ With just one catch
caught: MyException 5934185968946882193
Total time: 0.2883113300194964
------ With no catch
caught: MyException 4809429493926266912
<span class="sourceMarker">Total time: 0.29055844293907285</span>
</span></td></pre></tr></tbody></table></div>
<p>For the first two cases the result are around 11% better than the pure <code>runST</code>. Interestingly, for the third one <code>runSTthrowIO</code> still wins.</p>
<p>An even more drastic approach is to use <code>unsafeIOToST</code> and <code>unsafeSTToIO</code> in conjunction, modifying the original <code>fibMod</code>, allowing to freely insert the less pure <code>IO</code>-based <code>catch</code> while keeping it in <code>ST</code> only from an API's perspective. It's not entirely sound in terms of exception handling, but it is worth presenting.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">fibMod_H</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Int</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">ST</span> <span class="variable other generic-type haskell">s</span> <span class="storage type haskell">Int</span>
</span>fibMod_H n <span class="keyword operator haskell">=</span>
<span class="keyword control haskell">if</span> n <span class="keyword operator haskell"><</span> <span class="constant numeric integer decimal haskell">2</span>
<span class="keyword control haskell">then</span> return n
<span class="keyword control haskell">else</span> <span class="keyword control haskell">do</span> x <span class="keyword operator haskell"><-</span> newSTRef <span class="constant numeric integer decimal haskell">0</span>
y <span class="keyword operator haskell"><-</span> newSTRef <span class="constant numeric integer decimal haskell">1</span>
fibMod' n x y
<span class="keyword other haskell">where</span>
fibMod' <span class="constant numeric integer decimal haskell">0</span> x _ <span class="keyword operator haskell">=</span> readSTRef x
fibMod' n' x y <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
x' <span class="keyword operator haskell"><-</span> readSTRef x
y' <span class="keyword operator haskell"><-</span> readSTRef y
writeSTRef x y'
writeSTRef y <span class="keyword operator haskell">$!</span> x'<span class="keyword operator haskell">+</span>y'
when (n' <span class="keyword operator haskell">==</span> <span class="constant numeric integer decimal haskell">1000</span>) <span class="keyword operator haskell">$</span> <span class="keyword control haskell">do</span>
throw <span class="keyword operator haskell">$</span> <span class="constant other haskell">MyException</span> x'
<span class="keyword other haskell">let</span> recurse <span class="keyword operator haskell">=</span> fibMod' (n'<span class="keyword operator haskell">-</span><span class="constant numeric integer decimal haskell">1</span>) x y
<span class="keyword control haskell">if</span> n <span class="keyword operator haskell"><=</span> <span class="constant numeric integer decimal haskell">25000000</span>
<span class="keyword control haskell">then</span> recurse
<span class="keyword control haskell">else</span> <span class="sourceMarker">unsafeIOToST <span class="keyword operator haskell">$</span></span>
catch (<span class="sourceMarker">unsafeSTToIO</span> recurse)
(<span class="keyword operator haskell">\</span>(<span class="constant other haskell">MyException</span> _) <span class="keyword operator haskell">-></span> return (<span class="keyword operator haskell">-</span><span class="constant numeric integer decimal haskell">1</span>))
</span></td></pre></tr></tbody></table></div>
<p>With the prints:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"> putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>------ With lots of catches<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runST <span class="keyword operator haskell">$</span> fibMod_H <span class="constant numeric integer decimal haskell">50000000</span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="constant character escape haskell">\n</span>------ With just one catch<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runST <span class="keyword operator haskell">$</span> fibMod_H <span class="constant numeric integer decimal haskell">25000001</span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="constant character escape haskell">\n</span>------ With no catch<span class="punctuation definition string end haskell">"</span></span>
timeIt <span class="keyword operator haskell">$</span> print <span class="keyword operator haskell">$</span> runST <span class="keyword operator haskell">$</span> fibMod_H <span class="constant numeric integer decimal haskell">25000000</span>
</span></td></pre></tr></tbody></table></div>
<p>However, it does not improve on our cases:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">------ With lots of catches
-1
Total time: 3.401594614959322
------ With just one catch
-1
Total time: 1.2730798620032147
------ With no catch
caught: MyException 4809429493926266912
Total time: 0.30070900300052017
</span></td></pre></tr></tbody></table></div>
<p>The first two cases are considerably worse, and the third is still a bit better than using <code>STCatch</code>, but it's not a representative of the common case where <code>catch</code> is probably going to appear in the evaluation at least once. Final conclusion is that pure exceptions are still a win, if we wish to remain sound.</p>
Toward Better GHC Syntax Errors2015-04-06T00:00:00+00:002015-04-06T00:00:00+00:00https://blog.aloni.org/posts/toward-better-ghc-syntax-errors/<p>As a neophyte to functional programming and <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.haskell.org">Haskell<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, with full appreciation over the succinctly outputted type errors, for a long time I felt something was remiss with the output of syntax errors. In this post I present a preliminary fix to GHC that improves in the aforementioned arena.</p>
<span id="continue-reading"></span>
<p>Like many other compilers who have not yet forsaken the annotated BNF approach of describing a syntax and parser, Haskell's prominent compiler <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.haskell.org/ghc/">GHC<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> uses <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://www.haskell.org/happy/">Happy<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> in order to generate efficient parsing tables for the syntax.</p>
<h2 id="the-problem">The problem</h2>
<p>While there are some hand-coded hints about errors being detected while compiling a module with GHC, other errors sometimes don't provide any extra information.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 1 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
2
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell">test i <span class="keyword operator haskell">=</span> <span class="keyword other haskell">case</span> i <span class="keyword other haskell">of</span> <span class="constant numeric integer decimal haskell">2</span>
main <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 1 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example1.hs:2:1:
parse error (possibly incorrect indentation or mismatched brackets)
</span></td></pre></tr></tbody></table></div>
<p>And another example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 2 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
2
3
4
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell">test i <span class="keyword operator haskell">=</span> i
<span class="keyword other haskell">where</span> e
main <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 2 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example2.hs:4:1:
parse error (possibly incorrect indentation or mismatched brackets)
</span></td></pre></tr></tbody></table></div>
<p>As you can see, we have the same error on both occasions. Is it possible to generate an error message that differentiates between the two, automatically? Apparently yes.</p>
<h2 id="making-it-happier">Making it happier</h2>
<p>Thanks to the parsing tables, the parser does know which tokens can follow at the place of the error. So, I took the time to <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/da-x/happy/commits/master">modify Happy<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> and <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="https://github.com/da-x/ghc/commits/ghc-7.10-with-fixes">modify GHC<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> to produce the following results. The gist of these changes is to allow passing the list of <strong>next possible tokens</strong> to the user-provided error function. The results are not perfect, but they are interesting nonetheless.</p>
<p>Let's repeat the previous examples and add some more.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 1 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
2
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell">test i <span class="keyword operator haskell">=</span> <span class="keyword other haskell">case</span> i <span class="keyword other haskell">of</span> <span class="constant numeric integer decimal haskell">2</span>
main <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 1 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example1.hs:2:1:
parse error (possibly incorrect indentation or mismatched brackets), possible tokens: '|' '->'
</span></td></pre></tr></tbody></table></div>
<p>And for the second example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 2 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
2
3
4
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell">test i <span class="keyword operator haskell">=</span> i
<span class="keyword other haskell">where</span> e
main <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 2 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example2.hs:4:1:
parse error (possibly incorrect indentation or mismatched brackets), possible tokens: '=' '|'
</span></td></pre></tr></tbody></table></div>
<p>A third example:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 3 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
2
3
4
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell"><span class="keyword other haskell">data</span> <span class="constant other haskell">X</span> <span class="keyword operator haskell">=</span> <span class="constant other haskell">X</span> {
<span class="meta function type-declaration haskell"> <span class="entity name function haskell">bla</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Int</span>
</span><span class="meta function type-declaration haskell"> <span class="entity name function haskell">test</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Int</span>
</span> }
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 3 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example3.hs:3:10:
parse error on input, possible tokens: '}' ‘::’
</span></td></pre></tr></tbody></table></div>
<p>The avid reader would wonder at this point, why '::' is mentioned and ',' is not mentioned in the possible tokens list. We observe that the error is after 'Int test', probably much deeper and in a different syntax production than the one that defines each part of a record. With more time I would examine the <code>Action</code> and <code>Goto</code> tables generated by Happy from the extensive definition of the syntax in GHC.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 4 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
2
3
4
5
6
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell"><span class="keyword other haskell">data</span> <span class="constant other haskell">X</span> <span class="keyword operator haskell">=</span> <span class="constant other haskell">X</span> {
<span class="meta function type-declaration haskell"> <span class="entity name function haskell">bla</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">Int</span>
</span> <span class="punctuation separator comma haskell">,</span> test <span class="keyword operator haskell">::</span> <span class="constant other haskell">Int</span>
} <span class="keyword other haskell">deriving</span>
main <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 4 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example4.hs:6:1:
parse error (possibly incorrect indentation or mismatched brackets), possible tokens: '(' CONID QCONID PREFIXQCONSYM
</span></td></pre></tr></tbody></table></div>
<p>Here it gets more interesting, because the parser now outputs token identifiers that are not just punctuation or operators, but user-defined identifier strings. These are the same names used when defining the syntax.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 5 - source</td></tr><tr><td class="lineNumbers"><div class="lineNumbersDiv"><pre>1
</pre></div></rd><td class="sourceCode"><pre class="code"><span class="source haskell">main <span class="keyword operator haskell">=</span> <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div><div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">example 5 - output</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">example5.hs:1:8:
parse error on input, possible tokens: '_' 'as' 'case' 'do' 'hiding' 'if' 'let' 'qualified' 'forall' 'export' 'label' 'dynamic' 'safe' 'interruptible' 'unsafe' 'mdo' 'family' 'role' 'stdcall' 'ccall' 'capi' 'prim' 'javascript' 'proc' 'group' 'static' '{-# CORE' '{-# SCC' '{-# GENERATED' '\\' '~' '-' '[' '[:' '(' '(#' '(|' SIMPLEQUOTE VARID CONID QVARID QCONID PREFIXQVARSYM PREFIXQCONSYM IPDUPVARID CHAR STRING INTEGER RATIONAL PRIMCHAR PRIMSTRING PRIMINTEGER PRIMWORD PRIMFLOAT PRIMDOUBLE '[|' '[p|' '[t|' '[d|' '[||' TH_ID_SPLICE '$(' TH_ID_TY_SPLICE '$$(' TH_TY_QUOTE TH_QUASIQUOTE TH_QQUASIQUOTE ‘=’
</span></td></pre></tr></tbody></table></div>
<p>Obviously, there are many things that could come instead of the second <code>=</code>. Perhaps it would require some community debate regarding how to present this behemoth list of possible tokens to the user in this case.</p>
Build Maneuverings with External Linux Kernel Modules2014-12-12T00:00:00+00:002014-12-12T00:00:00+00:00https://blog.aloni.org/posts/build-maneuverings-with-external-linux-kernel-modules/<p>Much of the material relating to writing <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/Linux_kernel">Linux kernel<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> modules does not discuss the scenario where you would like to replace existing kernel code or a driver with a wrapping interface, or a whole new implementation of the same component, or another scenario where you have one external kernel module that depends on another. Our Linux kernel is a standalone component and it doesn't like these sort of tricks, but its build system is advanced enough to allow to implement them cleanly.</p>
<span id="continue-reading"></span><h2 id="first-just-a-plain-old-module">First, just a plain-old module</h2>
<p>To demonstrate the topics I am about to discuss, let's create a simple kernel module that we can work with:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">foo/Makefile</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source makefile"><span class="variable other makefile">foo-objs</span> <span class="keyword operator assignment makefile">+=</span></span> <span class="source makefile"><span class="meta string makefile"><span class="string unquoted makefile">foo-main.o</span></span>
<span class="variable other makefile">obj-m</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">foo.o</span></span>
</span></td></pre></tr></tbody></table></div>
<p>The following .gitignore file can be used:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">.gitignore</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">*.o
*.mod.o
*.ko
.*
modules.order
Module.symvers
</span></td></pre></tr></tbody></table></div>
<p>The C file can be very minimal for now:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">foo/foo-main.c</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source c"><span class="meta preprocessor include c"><span class="keyword control import include c">#include</span> <span class="string quoted other lt-gt include c"><span class="punctuation definition string begin c"><</span>linux/kernel.h<span class="punctuation definition string end c">></span></span>
</span>
<span class="storage type c">void</span> <span class="meta function c"><span class="entity name function c">foo_export</span></span><span class="meta function parameters c"><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function parameters c"><span class="meta group c"><span class="storage type c">void</span><span class="punctuation section group end c">)</span></span></span><span class="meta function c">
</span><span class="meta function c"><span class="meta block c"><span class="punctuation section block begin c">{</span></span></span><span class="meta function c"><span class="meta block c">
<span class="meta function-call c"><span class="variable function c">printk</span><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function-call c"><span class="meta group c">KERN_DEBUG <span class="string quoted double c"><span class="punctuation definition string begin c">"</span>foo_export called<span class="constant character escape c">\n</span><span class="punctuation definition string end c">"</span></span></span></span><span class="meta function-call c"><span class="meta group c"><span class="punctuation section group end c">)</span></span></span><span class="punctuation terminator c">;</span>
</span></span><span class="meta function c"><span class="meta block c"><span class="punctuation section block end c">}</span></span></span>
<span class="meta function-call c"><span class="variable function c">EXPORT_SYMBOL</span><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function-call c"><span class="meta group c">foo_export</span><span class="meta group c"><span class="punctuation section group end c">)</span></span></span><span class="punctuation terminator c">;</span>
</span></td></pre></tr></tbody></table></div>
<p>Building it for the currently running kernel is quite simple:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ make -C /lib/modules/`uname -r`/source M=`pwd`/foo
make: Entering directory `/usr/src/kernels/3.17.3-200.fc20.x86_64'
CC [M] /home/dan/module/foo/foo-main.o
LD [M] /home/dan/module/foo/foo.o
Building modules, stage 2.
MODPOST 1 modules
LD [M] /home/dan/module/foo/foo.ko
make: Leaving directory `/usr/src/kernels/3.17.3-200.fc20.x86_64'
</span></td></pre></tr></tbody></table></div>
<p>We should be able to load it via <code>insmod foo/foo.ko</code>. But, since the module does not do anything, and does not register on any subsystem, it is effectively just a library. A novice reader can add call to <code>foo_export</code> on module init. For now we will continue to focus on the building scriptology in this post.</p>
<h2 id="prepare-for-external-access">Prepare for external access</h2>
<p>We would like for another module to use our kernel code. But first, we need to export it via the standard C means, because <code>EXPORT_MODULE</code> is not enough - this just tells the kernel that it is okay to link against <code>foo_export</code> in module load time.</p>
<p>So we add this header under <code>foo/include</code>:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">foo/include/foo/foo.h</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source c"><span class="meta preprocessor c"><span class="keyword control import c">#ifndef</span> __FOO_MAIN_H__
</span><span class="meta preprocessor macro c"><span class="keyword control import define c">#define</span></span><span class="meta preprocessor macro c"> <span class="entity name constant preprocessor c">__FOO_MAIN_H__</span></span>
<span class="storage type c">void</span> <span class="meta function c"><span class="entity name function c">foo_export</span></span><span class="meta function parameters c"><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function parameters c"><span class="meta group c"><span class="storage type c">void</span><span class="punctuation section group end c">)</span></span></span><span class="punctuation terminator c">;</span>
<span class="meta preprocessor c"><span class="keyword control import c">#endif</span></span>
</span></td></pre></tr></tbody></table></div>
<p>We could have added a <code>foo.h</code> directly under <code>foo</code>, but that is not good because we would like to separate the interface from implementation. We expect other users to include the location of the header file via <code>-I</code>. Let's take care of our internal user <code>foo-main.c</code> too, by modifying the Makefile:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">foo/Makefile</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source makefile"><span class="meta group makefile"><span class="keyword control conditional makefile">ifneq</span> <span class="punctuation section group begin makefile">(</span><span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>LINUXINCLUDE<span class="keyword other block end makefile">}</span></span><span class="punctuation separator makefile">,</span><span class="punctuation section group end makefile">)</span></span>
<span class="comment line number-sign makefile"><span class="punctuation definition comment makefile">#</span> If we being invoked from kbuild, prepend the proper include paths
</span><span class="variable other makefile">LINUXINCLUDE</span> <span class="keyword operator assignment makefile">:=</span> <span class="meta string makefile"><span class="string unquoted makefile"><span class="punctuation separator continuation line makefile">\</span>
-I<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>M<span class="keyword other block end makefile">}</span></span>/include <span class="punctuation separator continuation line makefile">\</span>
<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>LINUXINCLUDE<span class="keyword other block end makefile">}</span></span></span></span>
<span class="keyword control conditional makefile">endif</span>
<span class="variable other makefile">foo-objs</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">foo-main.o</span></span>
<span class="variable other makefile">obj-m</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">foo.o</span></span>
</span></td></pre></tr></tbody></table></div>
<p>We can now proceed to also add <code>#include <foo/foo.h></code> in <code>foo-main.c</code>.</p>
<p><strong>NOTE:</strong> The <code>LINUXINCLUDE</code> directive in kbuild is very useful - it allows to insert new include paths before or after other paths or the kernel headers themselves. Here, we needed abstraction for our private module include paths. The module's code is now agnostic to where headers are located, which is a good preparation for any future point in time where the headers might move, and perhaps these headers can move into the kernel itself if our kernel code is really dandy and useful - who knows.</p>
<h2 id="a-second-dependent-kernel-module">A second, dependent kernel module</h2>
<p>Similarly to <code>foo</code>, we have created <code>bar</code>. However, in bar we insert a run-time dependency over foo:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">bar/bar-main.c</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source c"><span class="meta preprocessor include c"><span class="keyword control import include c">#include</span> <span class="string quoted other lt-gt include c"><span class="punctuation definition string begin c"><</span>linux/kernel.h<span class="punctuation definition string end c">></span></span>
</span><span class="meta preprocessor include c"><span class="keyword control import include c">#include</span> <span class="string quoted other lt-gt include c"><span class="punctuation definition string begin c"><</span>bar/bar.h<span class="punctuation definition string end c">></span></span>
</span><span class="meta preprocessor include c"><span class="keyword control import include c">#include</span> <span class="string quoted other lt-gt include c"><span class="punctuation definition string begin c"><</span>foo/foo.h<span class="punctuation definition string end c">></span></span>
</span>
<span class="storage type c">void</span> <span class="meta function c"><span class="entity name function c">bar_export</span></span><span class="meta function parameters c"><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function parameters c"><span class="meta group c"><span class="storage type c">void</span><span class="punctuation section group end c">)</span></span></span><span class="meta function c">
</span><span class="meta function c"><span class="meta block c"><span class="punctuation section block begin c">{</span></span></span><span class="meta function c"><span class="meta block c">
<span class="meta function-call c"><span class="variable function c">foo_export</span><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function-call c"></span><span class="meta function-call c"><span class="meta group c"><span class="punctuation section group end c">)</span></span></span><span class="punctuation terminator c">;</span>
<span class="meta function-call c"><span class="variable function c">printk</span><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function-call c"><span class="meta group c">KERN_DEBUG <span class="string quoted double c"><span class="punctuation definition string begin c">"</span>bar_export called<span class="constant character escape c">\n</span><span class="punctuation definition string end c">"</span></span></span></span><span class="meta function-call c"><span class="meta group c"><span class="punctuation section group end c">)</span></span></span><span class="punctuation terminator c">;</span>
</span></span><span class="meta function c"><span class="meta block c"><span class="punctuation section block end c">}</span></span></span>
<span class="meta function-call c"><span class="variable function c">EXPORT_SYMBOL</span><span class="meta group c"><span class="punctuation section group begin c">(</span></span></span><span class="meta function-call c"><span class="meta group c">bar_export</span><span class="meta group c"><span class="punctuation section group end c">)</span></span></span><span class="punctuation terminator c">;</span>
</span></td></pre></tr></tbody></table></div>
<p>But we would like to avoid the following error:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">/home/dan/test/bar/bar-main.c:2:21: fatal error: foo/foo.h: No such file or directory #include <foo/foo.h>
</span></td></pre></tr></tbody></table></div>
<p>Let's take the first step. We need to depend on <code>foo</code> in <code>bar</code>. If we would like to be completely flexible, we can support the modes where <code>foo</code> arrives from the kernel itself or from another external kernel module.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">bar/Makefile</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source makefile"><span class="meta group makefile"><span class="keyword control conditional makefile">ifneq</span> <span class="punctuation section group begin makefile">(</span><span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_PATH<span class="keyword other block end makefile">}</span></span><span class="punctuation separator makefile">,</span><span class="punctuation section group end makefile">)</span></span>
<span class="variable other makefile">FOO_INCLUDE</span><span class="keyword operator assignment makefile">=</span><span class="meta string makefile"><span class="string unquoted makefile">-I<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_PATH<span class="keyword other block end makefile">}</span></span>/include</span></span>
<span class="keyword control conditional makefile">endif</span>
<span class="meta group makefile"><span class="keyword control conditional makefile">ifneq</span> <span class="punctuation section group begin makefile">(</span><span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>LINUXINCLUDE<span class="keyword other block end makefile">}</span></span><span class="punctuation separator makefile">,</span><span class="punctuation section group end makefile">)</span></span>
<span class="comment line number-sign makefile"><span class="punctuation definition comment makefile">#</span> If we being invoked from kbuild, prepend the proper include paths
</span><span class="variable other makefile">LINUXINCLUDE</span> <span class="keyword operator assignment makefile">:=</span> <span class="meta string makefile"><span class="string unquoted makefile"><span class="punctuation separator continuation line makefile">\</span>
-I<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>M<span class="keyword other block end makefile">}</span></span>/include <span class="punctuation separator continuation line makefile">\</span>
<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_INCLUDE<span class="keyword other block end makefile">}</span></span> <span class="punctuation separator continuation line makefile">\</span>
<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>LINUXINCLUDE<span class="keyword other block end makefile">}</span></span></span></span>
<span class="keyword control conditional makefile">endif</span>
<span class="variable other makefile">bar-objs</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">bar-main.o</span></span>
<span class="variable other makefile">obj-m</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">bar.o</span></span>
</span></td></pre></tr></tbody></table></div>
<p>By adding <code>foo</code> to <code>LINUXINCLUDE</code> we can now have a working build, if we point <code>FOO_PATH</code> to the correct place when building <code>bar</code>.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">make -C /lib/modules/`uname -r`/source M=`pwd`/bar FOO_PATH=`pwd`/foo
</span></td></pre></tr></tbody></table></div>
<p>But is it not over yet, because we get an error from <code>modpost</code> (although the build is successful):</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">WARNING: "foo_export" [/home/dan/test/bar/bar.ko] undefined!
</span></td></pre></tr></tbody></table></div>
<p>The kernel keeps track of which modules are exported by which binary code. By default the program <code>modpost</code> that runs during the build process performs a lookup on all undefined symbols via the kernel's own <code>Module.symvers</code> located under <code>/lib/modules</code> in our case. <code>foo</code> is not there. Let's extend our kernel module's Makefile. Luckily we can pass <code>KBUILD_EXTRA_SYMBOLS</code> to kbuild, but it is ineffectual to do so from within the Makefile itself when it is invoked by kbuild.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="codeTitle" colspan="2">bar/Makefile</td></tr><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source makefile"><span class="meta group makefile"><span class="keyword control conditional makefile">ifneq</span> <span class="punctuation section group begin makefile">(</span><span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>LINUXINCLUDE<span class="keyword other block end makefile">}</span></span><span class="punctuation separator makefile">,</span><span class="punctuation section group end makefile">)</span></span>
<span class="comment line number-sign makefile"><span class="punctuation definition comment makefile">#</span> If we being invoked from kbuild, prepend the proper include paths
</span><span class="meta group makefile"><span class="keyword control conditional makefile">ifneq</span> <span class="punctuation section group begin makefile">(</span><span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_PATH<span class="keyword other block end makefile">}</span></span><span class="punctuation separator makefile">,</span><span class="punctuation section group end makefile">)</span></span>
<span class="variable other makefile">FOO_INCLUDE</span><span class="keyword operator assignment makefile">=</span><span class="meta string makefile"><span class="string unquoted makefile">-I<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_PATH<span class="keyword other block end makefile">}</span></span>/include</span></span>
<span class="keyword control conditional makefile">endif</span>
<span class="variable other makefile">LINUXINCLUDE</span> <span class="keyword operator assignment makefile">:=</span> <span class="meta string makefile"><span class="string unquoted makefile"><span class="punctuation separator continuation line makefile">\</span>
-I<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>M<span class="keyword other block end makefile">}</span></span>/include <span class="punctuation separator continuation line makefile">\</span>
<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_INCLUDE<span class="keyword other block end makefile">}</span></span> <span class="punctuation separator continuation line makefile">\</span>
<span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>LINUXINCLUDE<span class="keyword other block end makefile">}</span></span></span></span>
<span class="keyword control conditional makefile">else</span>
<span class="keyword control makefile">export</span><span class="variable other makefile"> FOO_PATH</span>
<span class="keyword control makefile">export</span><span class="variable other makefile"> KBUILD_EXTRA_SYMBOLS<span class="keyword operator assignment makefile">=</span></span><span class="meta string makefile"><span class="string unquoted makefile"><span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>FOO_PATH<span class="keyword other block end makefile">}</span></span>/Module.symvers</span></span>
<span class="meta function makefile"><span class="entity name function makefile">all</span></span><span class="keyword operator assignment makefile">:</span>
<span class="meta function arguments makefile"></span><span class="meta function body makefile"></span><span class="meta function body makefile"> <span class="source shell"><span class="meta function-call shell"><span class="variable function shell">make</span></span><span class="meta function-call arguments shell"><span class="variable parameter option shell"><span class="punctuation definition parameter shell"> -</span>C</span> <span class="variable parameter makefile"><span class="keyword other block begin makefile">${</span>KDIR<span class="keyword other block end makefile">}</span></span> M=<span class="keyword other block begin makefile">$(</span><span class="support function builtin makefile">shell</span> <span class="source shell"><span class="meta function-call shell"><span class="variable function shell">pwd</span></span></span><span class="keyword other block end makefile">)</span></span></span>
<span class="keyword control conditional makefile">endif</span>
</span><span class="variable other makefile">bar-objs</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">bar-main.o</span></span>
<span class="variable other makefile">obj-m</span> <span class="keyword operator assignment makefile">+=</span> <span class="meta string makefile"><span class="string unquoted makefile">bar.o</span></span>
</span></td></pre></tr></tbody></table></div>
<p>In order to not make the command line any longer, we have inserted a proxy target 'all'. Also, <code>KBUILD_EXTRA_SYMBOLS</code> and <code>FOO_PATH</code> are forwarded to kbuild, and <code>KDIR</code> receives the path of the kernel tree from the top level invocation.</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain">$ make -C bar KDIR=/lib/modules/`uname -r`/source FOO_PATH=`pwd`/foo
make: Entering directory `/home/dan/test/bar'
make -C /lib/modules/3.17.3-200.fc20.x86_64/source M=/home/dan/test/bar
make[1]: Entering directory `/usr/src/kernels/3.17.3-200.fc20.x86_64'
LD /home/dan/test/bar/built-in.o
CC [M] /home/dan/test/bar/bar-main.o
LD [M] /home/dan/test/bar/bar.o
Building modules, stage 2.
MODPOST 1 modules
CC /home/dan/test/bar/bar.mod.o
LD [M] /home/dan/test/bar/bar.ko
make[1]: Leaving directory `/usr/src/kernels/3.17.3-200.fc20.x86_64'
make: Leaving directory `/home/dan/test/bar'
</span></td></pre></tr></tbody></table></div>
<p>There are several things consider. <code>foo</code> must already be built when <code>bar</code> is built, otherwise <code>foo</code>'s <code>Module.symvers</code> is missing, and you would get a warning. The other thing to keep in mind is that the Makefile is evaluated twice - once in our direct execution and a second time when kbuild evaluates it. The current directory (`pwd`) in the second evaluation is the kernel tree, and it already contains a lots of useful bits such as <code>LINUXINCLUDE</code>. We should be careful about kbuild's own evaluation so that we don't accidentally override stuff that we did not intend or insert makefile targets that don't belong under the kbuild environment.</p>
<h1 id="careful-replacement">Careful replacement</h1>
<p>Note that the use of <code>LINUXINCLUDE</code> to replace existing kernel headers should be used with care. It is powerful enough to allow re-packaging of whole stacks of code that overlap with existing kernel software stack. Some kernel subsystems are spreading their headers over paths that don't necessarily start with <code>linux/</code>, for example, <code>uapi/linux</code> (user space headers), and <code>asm/</code> (architecture specific headers, actually located under <code>arch/*/include</code>). Ordering of paths when extending <code>LINUXINCLUDE</code> is key in that case.</p>
<p>One should mind including the proper headers so that the API are validated in compile time and match in run-time. For example, it is possible for an external package X to replace some modular existing kernel subsystem X with modified APIs, but when any other piece of code - user-space and kernel space - tries to compile against X, it should refer to the intended headers.</p>
Six Principles of How I Write My Journal2014-03-23T00:00:00+00:002014-03-23T00:00:00+00:00https://blog.aloni.org/posts/six-principles-of-how-i-write-my-journal/<p>Despite not being a frequent blogger, I am consistent with keeping a private journal that documents my life. The level of depth and introspection achievable with a private journal is different than what can be expected from a public blog or posts in a social network profile (which show mostly an external view), and over the years I have learned about how greatly it contributed to my decision making process and understanding of self.</p>
<span id="continue-reading"></span>
<p>There can be several advantages of keeping a private journal:</p>
<ul>
<li>Your writing pad behaves like a psychologist. Even if no one will ever read your journal, just writing about your life puts things in proper perspective. It's like a <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/Rubber_duck_debugging">Rubber duck debugging<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</li>
<li>Reading past journal entries teaches you about how much you changed and how you changed. Also, you find out that you have worried to much. Overall it would reduce your stress levels.</li>
<li>There are places you have been and people you have met and some decisions you have made that you would not like to forget many years down the road, because these understanding how your personality can thus far at any point in time can be helpful for knowing how it could develop even further.</li>
<li>If you ever want to spend countless hours some time in 2030 telling your kids the exact and glorious story of '<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/How_I_Met_Your_Mother">how I met your mother<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>' you'd be able to do so.</li>
<li>You might improve your writing skills if you don't have much of them to begin with.</li>
<li>It's your auto-biography material, in case you become that magnificent successful person you plan to be (don't we all need ambitions).</li>
</ul>
<p>It took a few years to find the best method to sustain a good private journal. I'd share you with some interdependent principles that I have garnered from the practice:</p>
<ol>
<li>Keep one entry per month. One month is not too short to make the journal seem too tedious and boring in retrospect, and not too long so you wouldn't forget the key aspects of how you spend your time, and your reflections over that spent time. Also, a month is the sweet spot that it takes most things to develop in a way that actual description of progress is possible.</li>
<li>Start writing the entry about a week before the month is due, and keep revising it until the 'deadline'. It is a drafting process which produces a nice and readable entry.</li>
<li>With that final week revisions you should find yourself making plans for the next month, and even writing them down, going back to see what you planned to do the previous month, etc.</li>
<li>Write more than 1,000 words. I usually reach 2,000. Turns out your life is less boring than you thought.</li>
<li>Pick a title for each entry, like a chapter from a book. And add a soundtrack. Just kidding.</li>
<li>Write about everything, encrypt the damn thing, and don't let anyone read it, including the NSA (I don't back it up to the internet anywhere conspicuously). Otherwise, you would not write honest entries.</li>
</ol>
<p>If you do that for 50 years, you would get one hell of a book. That's the book of your life. And whether you decide to use it exclusively for reminiscing by taking it to your grave, or be careless about it near death and open it up for any interested party, it would be worthwhile.</p>
Extending Monads for Debugging in Haskell2012-07-02T00:00:00+00:002012-07-02T00:00:00+00:00https://blog.aloni.org/posts/extending-monads-for-debugging-in-haskell/<p>One of the nice things about <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.haskell.org/haskellwiki/Haskell">Haskell<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> is the ability to extend the class of <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.haskell.org/tutorial/monads.html">Monads<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>.</p>
<span id="continue-reading"></span>
<p>One of the original purposes of Monads was to describe flow while leaving the implementation of the flow to a later stage. This allows to define what happens as a side effect of the computational steps.</p>
<p>For example, let's say we have a computation that we would like to debug. If we formulate it algebraically, it would be harder to checks the step of the computation in run-time. So naturally we break the computation into statement-like stages, introducing code to trace the intermediate results in between.</p>
<p>However, sometimes we would also like to <strong>keep the performance of the 'untraced' computation as it was</strong>. In C++ we can can use templates in order to instantiate two implementations of the computation. In C we would probably use macros trickery of some sort along with static inline functions that define to nothing. In Python we would probably use a global debug variable, the <code>__debug__</code> builtin, or a dedicated logging library.</p>
<p>In Haskell, this can come naturally as an extension of the Monad class, with the advantage of multiple instantiation. To illustrate, let's extend Monad with MonadDebug:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta import haskell"><span class="keyword other haskell">import</span> <span class="support other module haskell">Control.Monad.Identity</span> <span class="meta declaration exports haskell">(<span class="entity name function haskell">runIdentity</span><span class="punctuation separator comma haskell">,</span> <span class="storage type haskell">Identity</span>)</span></span>
<span class="meta declaration class haskell"><span class="keyword other haskell">class</span> <span class="support class prelude haskell">Monad</span> <span class="variable other generic-type haskell">m</span> => <span class="entity other inherited-class haskell">MonadDebug</span> <span class="variable other generic-type haskell">m</span> <span class="keyword other haskell">where</span></span>
<span class="meta function type-declaration haskell"> <span class="entity name function haskell">logDebug</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">String</span> <span class="keyword other arrow haskell">-></span> <span class="variable other generic-type haskell">m</span> <span class="support constant unit haskell">()</span>
</span><span class="meta declaration instance haskell"><span class="keyword other haskell">instance</span> <span class="storage type haskell">MonadDebug</span> <span class="storage type haskell">IO</span> <span class="keyword other haskell">where</span></span>
logDebug s <span class="keyword operator haskell">=</span> putStrLn s
<span class="meta declaration instance haskell"><span class="keyword other haskell">instance</span> <span class="storage type haskell">MonadDebug</span> <span class="storage type haskell">Identity</span> <span class="keyword other haskell">where</span></span>
logDebug _ <span class="keyword operator haskell">=</span> return <span class="constant language unit haskell">()</span>
</span></td></pre></tr></tbody></table></div>
<p>Those instances make it possible to use class function <code>logDebug</code> directly under IO or under pure computations with the Identity Monad.</p>
<p>Let's define a sample computation function:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">computation</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">MonadDebug</span> <span class="variable other generic-type haskell">m</span> <span class="keyword other big-arrow haskell">=></span> <span class="storage type haskell">Integer</span> <span class="keyword other arrow haskell">-></span> <span class="storage type haskell">Integer</span> <span class="keyword other arrow haskell">-></span> <span class="variable other generic-type haskell">m</span> <span class="storage type haskell">Integer</span>
</span>computation x y <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
<span class="keyword other haskell">let</span> t1 <span class="keyword operator haskell">=</span> x * <span class="constant numeric integer decimal haskell">2</span> <span class="keyword operator haskell">+</span> y
logDebug <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Here, t1=<span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> (show t1)
<span class="keyword other haskell">let</span> t2 <span class="keyword operator haskell">=</span> t1 * <span class="constant numeric integer decimal haskell">3</span> <span class="keyword operator haskell">+</span> x
logDebug <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Here, t2=<span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> (show t2)
<span class="keyword other haskell">let</span> t3 <span class="keyword operator haskell">=</span> t2 * <span class="constant numeric integer decimal haskell">7</span> <span class="keyword operator haskell">-</span> x * x
logDebug <span class="keyword operator haskell">$</span> <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Here, t3=<span class="punctuation definition string end haskell">"</span></span> <span class="keyword operator haskell">++</span> (show t3)
return t3
</span></td></pre></tr></tbody></table></div>
<p>The type signature for <code>computation</code> is optional, and can be inferred by the compiler, simply because we referenced logDebug under our Monad.</p>
<p>Now, let's try to use it under the two environments. Here's the code:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="source haskell"><span class="meta function type-declaration haskell"><span class="entity name function haskell">main</span> <span class="keyword other double-colon haskell">::</span> <span class="storage type haskell">IO</span> <span class="support constant unit haskell">()</span>
</span>main <span class="keyword operator haskell">=</span> <span class="keyword control haskell">do</span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Run computation as pure:<span class="punctuation definition string end haskell">"</span></span>
<span class="keyword other haskell">let</span> t <span class="keyword operator haskell">=</span> runIdentity <span class="keyword operator haskell">$</span> computation <span class="constant numeric integer decimal haskell">1</span> <span class="constant numeric integer decimal haskell">2</span>
putStr <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Result: <span class="punctuation definition string end haskell">"</span></span>
print t
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span><span class="punctuation definition string end haskell">"</span></span>
putStrLn <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Run computation with impure IO logging stages: <span class="punctuation definition string end haskell">"</span></span>
t' <span class="keyword operator haskell"><-</span> computation <span class="constant numeric integer decimal haskell">1</span> <span class="constant numeric integer decimal haskell">2</span>
putStr <span class="string quoted double haskell"><span class="punctuation definition string begin haskell">"</span>Result: <span class="punctuation definition string end haskell">"</span></span>
print t'
</span></td></pre></tr></tbody></table></div>
<p>Let's try to run it:</p>
<div class="mycode-block"><table class="codeBox"><tbody><tr><td class="sourceCode sourceCodeWrap"><pre class="code"><span class="text plain"># runghc test.hs
Run computation as pure:
Result: 90
Run computation with impure IO logging stages:
Here, t1=4
Here, t2=13
Here, t3=90
Result: 90
</span></td></pre></tr></tbody></table></div>
<p>The advantage is that the compiler can optimize the pure Identity Monad computation much better compared to the non-pure computation, and we achieve this without using any Haskell constructs that are much sophisticated.</p>
<p>p.s. A novice reader might also be able to devise MonadDebug instances for the various Monad transformer classes under the cases where MonadIO is the underlying Monad.</p>
Success of VM infrastructure explained by historically crippled OS design2010-12-22T00:00:00+00:002010-12-22T00:00:00+00:00https://blog.aloni.org/posts/success-of-vm-infrastructure-explained-by-historically-crippled-os-design/<p>During the last decade we have seen the rise of server and desktop virtualisation infrastructure as the official and standard means of creating services and resources isolation at both the client and server side. System virtualization provided rigid management of computing resources over standardized PC and server hardware for the first time.</p>
<span id="continue-reading"></span>
<p>However when one thinks about it from an engineering perspective - why does a full PC system virtualization needed in order to achieve 'rigid management of computing resources'? Why couldn't <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/Operating_system">OS<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> designers create those abilities in the first place? Why do we need hyper-visors such as <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.xen.org/">Xen<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> and software such as one provided by <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.vmware.com">VMware<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> in order to attain these abilities?</p>
<p>Short answer is that all major OS writers did not provide the proper infrastructure for running sub-instances of the OS <strong>efficiently</strong> and managing those instances with Quality of Service and fail-over on a network of hardware. A standard OS only provides the means of running unmigratable processes. A standard OS is there just to provide isolation between processes, scheduling, memory management, hardware abstraction, implementation of file systems and networking protocols. In the past there were attempts to extend the OS concept in order to accommodate clustering capabilities.</p>
<p>Linux serves as a good ground for these kind of experiments. Take for example - <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.mosix.org">MOSIX<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, which is a project that extends Linux in a sense that one Linux system encompasses more than one PC, allowing for process migration. However process migration on its own is inadequate since a process is just one part that composes 'service', and you would actually want to migrate whole services efficiently in a fully isolated manner. Another project is '<span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://user-mode-linux.sourceforge.net">User Mode Linux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>' (often confused with <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://en.wikipedia.org/wiki/Unified_Modeling_Language">UML<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> which is something else), which allows running the Linux kernel as a Linux process. At some point in time there was a resource project that added suspend/resume support to 'User Mode Linux', but except that ability, 'User Model Linux' did not gain ground in the virtualization field.</p>
<p>Take another example - <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.colinux.org">Cooperative Linux<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span>, written by yours truly. Except for helping lots of Windows users fiddling with Linux on their desktop machines, it did not gain ground in the virtualization industry on servers (even though it also runs on Linux, with the best opportunity for performance). All those projects are just partial hacks in the direction of VM infrastructure and none seem to provide it in a productive sense. All major OS designs are crippled in a sense that you need to virtualized them (i.e. put them in "a box") in order to attain VM infrastructure capabilities. Key for VM infrastructure is that you must provide the ability to manage your VMs on a cluster in an enterprise level manner and capabilities. Of course, I don't blame OS writers for not detecting these needs in the corporate world. Actually, there were a few engineers that did detect it, and even awhile back. IBM already <span class='external-link-outer'><a class="external-link"rel="noopener" target="_blank" href="http://www.networkworld.com/news/2009/043009-ibm-virtualization.html">came up with the idea<span class='external-link-no-underline'></span><svg width='13.5' height='13.5' aria-hidden='true' viewBox='0 0 24 24' class='iconExternalLink_wgqa'><path fill='currentColor' d='M21 13v10h-21v-19h12v2h-10v15h17v-8h2zm3-12h-10.988l4.035 4-6.977 7.07 2.828 2.828 6.977-7.07 4.125 4.172v-11z'></path></svg></a></span> of virtualization awhile back on their mainframes. However, that was the beginning - and later on when traditional OS designs for single, desktop users emerged those features were not needed. Unfortunately those OSes also propagated to the server side, and only few saw the potential to bring the virtualization feature back under the cloth of the 'full system virtualization' method, adding a VM infrastructure.</p>
<p>Rethinking the Operating System concept, one would come up with a '<strong>Super Operating System</strong>' concept (i.e. SOS, also a funny overlap with the acronym for - "Save Our Souls"), that would define a new Operating System from the ground up that would also conform to all the attributes that are currently provided by standard 'VM infrastructure'. SOS should be able to run instances of itself, connecting those instances with their dependencies. For example, I would be able to run two instances and connect them with a virtual Ethernet link (of course, I'd need a separate network stack for each instance). It should also make it possible to migrate its sub-instances, and other aspects of current-day's VM management.</p>
<p>I know that designing a new OS as a standalone mission is somewhat unessential, but I have other ideas to pursue in that venue that would might improve the software world a little, but that's a topic for another post.</p>