crier

test.atom.xml (442542B)

---

 <?xml version="1.0" encoding="utf-8"?>
 <feed xmlns="http://www.w3.org/2005/Atom"><title>man bytes gnu</title><link href="/" rel="alternate"></link><link href="feeds/all.atom.xml" rel="self"></link><id>/</id><updated>2024-07-07T21:03:40+02:00</updated><entry><title>Support Your Local Viewer</title><link href="local-markdown.html" rel="alternate"></link><published>2024-07-07T21:03:40+02:00</published><updated>2024-07-07T21:03:40+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2024-07-07:local-markdown.html</id><summary type="html">&lt;p class="first last"&gt;Bash script to render and spawn a viewer for markdown files&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;Markdown is the fast-food of document formats.&lt;/p&gt;
 &lt;p&gt;That doesn't change the fact that it's everywhere.&lt;/p&gt;
 &lt;p&gt;So much everywhere, in fact, that it's kind of puzzling there is not a dedicated tool around to view it.&lt;/p&gt;
 &lt;div class="section" id="pinning-down-markdown"&gt;
 &lt;h2&gt;Pinning down markdown&lt;/h2&gt;
 &lt;p&gt;There is no shortage of applications that &lt;em&gt;can&lt;/em&gt; render markdown. Among the alternatives are free code editors like &lt;a class="reference external" href="https://atom-editor.cc/"&gt;Atom&lt;/a&gt; or &lt;a class="reference external" href="https://plugins.geany.org/markdown.html"&gt;Geany&lt;/a&gt;, the browser plugin &lt;a class="reference external" href="https://chromewebstore.google.com/detail/markdown-viewer/ckkdlimhmcjmikdlpkmbgfkaikojcbjk"&gt;Markdown Viewer&lt;/a&gt; and even a dedicated markdown editor like &lt;a class="reference external" href="https://github.com/marktext/marktext"&gt;Marktext&lt;/a&gt;, &lt;a class="reference external" href="https://github.com/retext-project/retext"&gt;retext&lt;/a&gt; and &lt;a class="reference external" href="https://ghostwriter.kde.org/"&gt;ghostwriter&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;And of course, there exist SaaS offerings such as &lt;a class="reference external" href="https://hackmd.io"&gt;hackmd&lt;/a&gt;. But seeing as those are not alternatives for offline use, we don't concern ourselves with those here.&lt;/p&gt;
 &lt;p&gt;But what is the equivalent of &lt;a class="reference external" href="https://github.com/muennich/sxiv"&gt;sxiv&lt;/a&gt; or &lt;a class="reference external" href="https://feh.finalrewind.org"&gt;feh&lt;/a&gt; for Markdown?&lt;/p&gt;
 &lt;p&gt;Honestly, I couldn't find any such thing. If there is, I'd &lt;a class="reference external" href="https://holbrook.no/msg"&gt;love to know&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Fortunately, there is a perfectly reasonable workaround.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="step-by-step"&gt;
 &lt;h2&gt;Step by step&lt;/h2&gt;
 &lt;p&gt;After all, it is in the spirit of *nixes to use a choice of tools who &lt;em&gt;does one thing and does it well&lt;/em&gt;.&lt;/p&gt;
 &lt;p&gt;So, no matter how bizarre it feels, it may make sense that a lurid format like &lt;em&gt;Markdown&lt;/em&gt; should be treated in a separate step to produce a more well-established - and less ambiguous - format.&lt;/p&gt;
 &lt;p&gt;I asked a related question on &lt;a class="reference external" href="https://tex.stackexchange.com/questions/341899/latex-to-markdown-converter"&gt;Stackexchange&lt;/a&gt; long ago, and there the &lt;tt class="docutils literal"&gt;pandoc&lt;/tt&gt; tool came up as a solution.&lt;/p&gt;
 &lt;p&gt;And it turns out it works beautifully in this case aswell.&lt;/p&gt;
 &lt;p&gt;Consider the following script:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;--suffix&lt;span class="o"&gt;=&lt;/span&gt;.html&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 pandoc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;gfm&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;html&lt;span class="w"&gt; &lt;/span&gt;-M&lt;span class="w"&gt; &lt;/span&gt;document-css&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--standalone&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/null
 w3m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Quite simply, we generate a standalone html file in &lt;tt class="docutils literal"&gt;tmpfs&lt;/tt&gt;, which in turn is read and renderered by a web browser.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="browsing-browsers"&gt;
 &lt;h2&gt;Browsing browsers&lt;/h2&gt;
 &lt;p&gt;No appreciation for &lt;a class="reference external" href="https://w3m.sourceforge.net"&gt;w3m&lt;/a&gt;, eh? Instead want that fuzzy feel of comforting colors, smooth scroll and fancy fonts?&lt;/p&gt;
 &lt;p&gt;I can understand. I used to suffer that affliction, too.&lt;/p&gt;
 &lt;p&gt;But nonetheless; it's an important point. For example, what if I wanted to use &lt;a class="reference external" href="https://lynx.browser.org"&gt;lynx&lt;/a&gt; or &lt;a class="reference external" href="https://fanglingsu.github.io/vimb"&gt;vimb&lt;/a&gt; instead? Choosing the browser to view with should definitely be the caller's call.&lt;/p&gt;
 &lt;p&gt;Is there a canonical way of doing that in Linux.&lt;/p&gt;
 &lt;p&gt;Kind of.&lt;/p&gt;
 &lt;p&gt;Let's review a couple of the options.&lt;/p&gt;
 &lt;div class="section" id="the-environmental-solution"&gt;
 &lt;h3&gt;The environmental solution&lt;/h3&gt;
 &lt;p&gt;Some applications honor the &lt;tt class="docutils literal"&gt;$BROWSER&lt;/tt&gt; environment variable. So let's cover for that:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;browser&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;w3m&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
 &lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;--suffix&lt;span class="o"&gt;=&lt;/span&gt;.html&lt;span class="k"&gt;)&lt;/span&gt;
 pandoc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;gfm&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;html&lt;span class="w"&gt; &lt;/span&gt;-M&lt;span class="w"&gt; &lt;/span&gt;document-css&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--standalone&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/null
 &lt;span class="nv"&gt;$browser&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Now, viewing the markdown file &lt;tt class="docutils literal"&gt;README.md&lt;/tt&gt; with &lt;tt class="docutils literal"&gt;lynx&lt;/tt&gt; is as easy as:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;which&lt;span class="w"&gt; &lt;/span&gt;lynx&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bash&lt;span class="w"&gt; &lt;/span&gt;wmd.sh&lt;span class="w"&gt; &lt;/span&gt;README.md
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-cross-solution"&gt;
 &lt;h3&gt;The cross solution&lt;/h3&gt;
 &lt;p&gt;So, have you heard about the Cross Desktop Group? &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt; These are the guys you should be sending a thought of gratitude every time you intuitively look in your &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;~/.config&lt;/span&gt;&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;~/.local&lt;/span&gt;&lt;/tt&gt; folder for application data. And if you're a &lt;strong&gt;HUIf&lt;/strong&gt; &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt; coder, chances are you have seen the string &lt;tt class="docutils literal"&gt;xdg&lt;/tt&gt; in some function call somewhere.&lt;/p&gt;
 &lt;p&gt;These days they go by the name &lt;a class="reference external" href="https://www.freedesktop.org/"&gt;Free Desktop Group&lt;/a&gt;, and among other things they have negotiated something particularly useful to us in this particular case.&lt;/p&gt;
 &lt;p&gt;To launch programs in a desktop environment in Linux, a &lt;a class="reference external" href="https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html"&gt;Desktop Entry Specification&lt;/a&gt; file format &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt; has been defined - appropriately suffixed &lt;tt class="docutils literal"&gt;.desktop&lt;/tt&gt;.&lt;/p&gt;
 &lt;p&gt;Take a look at a random &lt;tt class="docutils literal"&gt;*.desktop&lt;/tt&gt; file in &lt;tt class="docutils literal"&gt;/usr/share/applications&lt;/tt&gt; (your most likely default &lt;tt class="docutils literal"&gt;$XDG_DATA_DIRS&lt;/tt&gt; location, where xdg searches for desktop files). Every one will contain an top-level &lt;tt class="docutils literal"&gt;Exec=&lt;/tt&gt; entry.&lt;/p&gt;
 &lt;p&gt;For example, my &lt;tt class="docutils literal"&gt;feh.desktop&lt;/tt&gt; entry has &lt;tt class="docutils literal"&gt;Exec=feh &lt;span class="pre"&gt;--start-at&lt;/span&gt; %u&lt;/tt&gt; which means find &lt;tt class="docutils literal"&gt;feh&lt;/tt&gt; in &lt;tt class="docutils literal"&gt;$PATH&lt;/tt&gt; and execute it with the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;--start-at&lt;/span&gt;&lt;/tt&gt; switch and one single &lt;em&gt;url&lt;/em&gt; as argument:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;[Desktop Entry]&lt;/span&gt;
 &lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Feh&lt;/span&gt;
 &lt;span class="na"&gt;Name[en_US]&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;feh&lt;/span&gt;
 &lt;span class="na"&gt;GenericName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Image viewer&lt;/span&gt;
 &lt;span class="na"&gt;GenericName[en_US]&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Image viewer&lt;/span&gt;
 &lt;span class="na"&gt;Comment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Image viewer and cataloguer&lt;/span&gt;
 &lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;feh --start-at %u&lt;/span&gt;
 &lt;span class="na"&gt;Terminal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;
 &lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Application&lt;/span&gt;
 &lt;span class="na"&gt;Icon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;feh&lt;/span&gt;
 &lt;span class="na"&gt;Categories&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Graphics&lt;/span&gt;&lt;span class="c1"&gt;;2DGraphics;Viewer;&lt;/span&gt;
 &lt;span class="na"&gt;MimeType&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;image/bmp&lt;/span&gt;&lt;span class="c1"&gt;;image/gif;image/jpeg;image/jpg;image/pjpeg;image/png;image/tiff;image/webp;image/x-bmp;image/x-pcx;image/x-png;image/x-portable-anymap;image/x-portable-bitmap;image/x-portable-graymap;image/x-portable-pixmap;image/x-tga;image/x-xbitmap;image/heic;&lt;/span&gt;
 &lt;span class="na"&gt;NoDisplay&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Covering this case in our script:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;fallbackbrowsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;w3m
 &lt;span class="nv"&gt;browsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;
 &lt;span class="c1"&gt;# if browser env exists, then&lt;/span&gt;
 &lt;span class="c1"&gt;# try handle it as a desktop entry&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# find the xdg paths&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;XDG_DATA_DIRS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;XDG_DATA_DIRS&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="p"&gt;/usr/share&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_DATA_HOME&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nv"&gt;XDG_DATA_DIRS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$XDG_DATA_HOME&lt;/span&gt;:&lt;span class="nv"&gt;$XDG_DATA_DIRS&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# split on &amp;quot;:&amp;quot; and try dirs one by one&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# use first matching browser desktop entry&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;_ifs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$IFS&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;:
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;dirs&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_DATA_DIRS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$dirs&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nv"&gt;s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;.desktop
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/applications/&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;.desktop
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$a&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;browsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;gtk-launch &lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;
 &lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="c1"&gt;# if no browser set or could not be found in xdg,&lt;/span&gt;
 &lt;span class="c1"&gt;# then try the browser env var as command, or&lt;/span&gt;
 &lt;span class="c1"&gt;# ultimately the static fallback&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$browsercmd&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;browsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;$fallbackbrowsercmd&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
 &lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;--suffix&lt;span class="o"&gt;=&lt;/span&gt;.html&lt;span class="k"&gt;)&lt;/span&gt;
 pandoc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;gfm&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;html&lt;span class="w"&gt; &lt;/span&gt;-M&lt;span class="w"&gt; &lt;/span&gt;document-css&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--standalone&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/null
 &lt;span class="nv"&gt;$browsercmd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-default-solution"&gt;
 &lt;h3&gt;The default solution&lt;/h3&gt;
 &lt;p&gt;Yes, there is such a thing as &amp;quot;default web browser&amp;quot; in &lt;tt class="docutils literal"&gt;XDG&lt;/tt&gt;, too.&lt;/p&gt;
 &lt;p&gt;Have a look at &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;xdg-settings&lt;/span&gt; &lt;span class="pre"&gt;--list&lt;/span&gt;&lt;/tt&gt;. On my system, it shows a rather modest output:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;xdg-settings&lt;span class="w"&gt; &lt;/span&gt;--list
 &lt;span class="go"&gt;Known properties:&lt;/span&gt;
 &lt;span class="go"&gt;        default-url-scheme-handler    Default handler for URL scheme&lt;/span&gt;
 &lt;span class="go"&gt;        default-web-browser           Default web browser&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;And the default web browser is:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;xdg-settings&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;default-web-browser
 &lt;span class="go"&gt;brave-browser.desktop&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Yeah, yeah, yeah. Busted. I use graphical browsers, too.&lt;/p&gt;
 &lt;p&gt;Now let's add this to the mix, then.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;fallbackbrowsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;w3m
 &lt;span class="nv"&gt;browsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;
 &lt;span class="c1"&gt;# if browser env var not set, set it with default browser&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;xdg-settings&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;default-web-browser&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="p"&gt;%%.*&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
 &lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;XDG_DATA_DIRS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;XDG_DATA_DIRS&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="p"&gt;/usr/share&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_DATA_HOME&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nv"&gt;XDG_DATA_DIRS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$XDG_DATA_HOME&lt;/span&gt;:&lt;span class="nv"&gt;$XDG_DATA_DIRS&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;_ifs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$IFS&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;:
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;dirs&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_DATA_DIRS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$dirs&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nv"&gt;s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;.desktop
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nv"&gt;a&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/applications/&lt;span class="nv"&gt;$BROWSER&lt;/span&gt;.desktop
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$a&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;browsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;gtk-launch &lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;
 &lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$browsercmd&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;browsercmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;$fallbackbrowsercmd&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
 &lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;--suffix&lt;span class="o"&gt;=&lt;/span&gt;.html&lt;span class="k"&gt;)&lt;/span&gt;
 pandoc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;gfm&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;html&lt;span class="w"&gt; &lt;/span&gt;-M&lt;span class="w"&gt; &lt;/span&gt;document-css&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--standalone&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/null
 &lt;span class="nv"&gt;$browsercmd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="naming-the-executioner"&gt;
 &lt;h2&gt;Naming the executioner&lt;/h2&gt;
 &lt;p&gt;So now we have a markdown viewer. And it can be as lean or as heavy as you want it to be. It's all to the browser you choose.&lt;/p&gt;
 &lt;p&gt;To make it accessible, map the script as a command alias in your &lt;tt class="docutils literal"&gt;.profile&lt;/tt&gt; or &lt;tt class="docutils literal"&gt;.bashrc&lt;/tt&gt; and you have the viewer at your fingertips.&lt;/p&gt;
 &lt;p&gt;I call mine &lt;tt class="docutils literal"&gt;wmd&lt;/tt&gt;:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;alias&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;wmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bash &lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/scripts/markdown.sh&amp;quot;&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;And voilá:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;BROWSER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lynx
 &lt;span class="gp"&gt;$ &lt;/span&gt;wmd&lt;span class="w"&gt; &lt;/span&gt;README.md
 &lt;/pre&gt;&lt;/div&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Or X Desktop Group, as they were originally called.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Human User Interface. Don't bother looking - I made it up. As the cause of AI, robots and machines inevitably will become appropriated by the woke intersectionality complex, the terminology is probably going to need such distictions.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Actually the format is &lt;a class="reference external" href="https://en.wikipedia.org/wiki/INI_file"&gt;ini&lt;/a&gt;. The standard is rather in the file naming, really.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Offlining"></category><category term="bash"></category><category term="markdown"></category><category term="pandoc"></category><category term="vimb"></category><category term="w3m"></category><category term="lynx"></category><category term="xdg"></category></entry><entry><title>Secrets in the shell</title><link href="clortho.html" rel="alternate"></link><published>2024-06-25T20:46:00+02:00</published><updated>2024-06-25T21:58:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2024-06-25:clortho.html</id><summary type="html">&lt;p class="first last"&gt;A key value store at your fingertips&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;Ever since I started using the &lt;a class="reference external" href="https://www.passwordstore.org/"&gt;pass&lt;/a&gt; CLI as my password manager, I've found myself putting all sorts of stuff in there; usernames, email, urls, crypto addresses, api keys, you name it.&lt;/p&gt;
 &lt;p&gt;It makes total sense that some of these items are in there. For example, I store the url to a service together with the password, usually accompanied by the username and the email used &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;. No password recoveries needed.&lt;/p&gt;
 &lt;p&gt;However, at some point I also started putting in things like crypto addresses, or even token smart contract addresses in there.&lt;/p&gt;
 &lt;p&gt;That seemed less of a good fit.&lt;/p&gt;
 &lt;p&gt;One thing is that it spams the password directory.&lt;/p&gt;
 &lt;p&gt;Another, perhaps more sinister, issue is that it's pretty clear for anyone reading the directory what items you are storing data for.&lt;/p&gt;
 &lt;p&gt;So what if I want to store key/value pairs, and at the same time I want to hide what I am storing?&lt;/p&gt;
 &lt;div class="section" id="hiding-in-plain-cipher-text"&gt;
 &lt;h2&gt;Hiding in plain cipher text&lt;/h2&gt;
 &lt;p&gt;A simple solution is to hash the key together with some passphrase as &amp;quot;salt&amp;quot;, along the lines of this:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;
 &lt;span class="c1"&gt;# read passphrase from stdin&lt;/span&gt;
 stty&lt;span class="w"&gt; &lt;/span&gt;-echo
 &lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;passphrase: &amp;quot;&lt;/span&gt;
 &lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;passphrase
 stty&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;
 hash_key&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# dump the passphrase (assuming mktemp stores in volatile memory)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;ktt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;kt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;200&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kt&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passphrase&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kt&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;600&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kt&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# the salted key; first add the hashed passphrase (sha512)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;kc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;sha512sum&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{print $1;}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ktt&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# remove the stored passphrase&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;shred&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kt&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# then add the key to the mix, and hash again&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ktt&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;kc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;sha512sum&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ktt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{print $1;}&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="o"&gt;}&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;The &lt;tt class="docutils literal"&gt;kc&lt;/tt&gt; variable at the end of the script above now contains the &lt;em&gt;salted&lt;/em&gt; hash of the key.&lt;/p&gt;
 &lt;p&gt;Specifically, in this implementation, given key &lt;tt class="docutils literal"&gt;&amp;quot;foo&amp;quot;&lt;/tt&gt; and passphrase &lt;tt class="docutils literal"&gt;&amp;quot;bar&amp;quot;&lt;/tt&gt;, the resulting key will be:&lt;/p&gt;
 &lt;p&gt;&lt;tt class="docutils literal"&gt;a9454592edbed78c3abb739dd88567bc92cc4ea1feee8480c7204d48d7d0e9ce86f9c79d55e39751ceb3f2774913841056293a5e8e8f440a3f281dffabd6f540&lt;/tt&gt;.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="added-value"&gt;
 &lt;h2&gt;Added value&lt;/h2&gt;
 &lt;p&gt;Now we can encrypt the store the corresponding value, and store it in a file that has that literal hash as the key.&lt;/p&gt;
 &lt;p&gt;Using &lt;a class="reference external" href="https://ccrypt.sourceforge.net/"&gt;ccrypt&lt;/a&gt; as an example:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;data_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/.obfuscated
 &lt;span class="nv"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;
 &lt;span class="nv"&gt;vp&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;
 &lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$vp&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 &lt;span class="c1"&gt;# create a file for ccrypt to read, because supplying any other way in shell is not so safe.&lt;/span&gt;
 &lt;span class="nv"&gt;passfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="k"&gt;)&lt;/span&gt;
 chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;600&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passfile&lt;/span&gt;
 &lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passphrase&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passfile&lt;/span&gt;
 chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;400&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passfile&lt;/span&gt;
 ccrypt&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passfile&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;
 shred&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$passfile&lt;/span&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;key&lt;span class="w"&gt; &lt;/span&gt;fail
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
 &lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;span class="c1"&gt;# run the code in the previous section&lt;/span&gt;
 hash_key
 mkdir&lt;span class="w"&gt; &lt;/span&gt;-vp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$data_dir&lt;/span&gt;
 cp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;.cpt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$data_dir&lt;/span&gt;/&lt;span class="nv"&gt;$kc&lt;/span&gt;
 shred&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;.cpt
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;At the end of this operation, the &lt;em&gt;encrypted&lt;/em&gt; value is stored in a file whose name is the &lt;em&gt;salted&lt;/em&gt; hash of the key.&lt;/p&gt;
 &lt;div class="section" id="key-mastering"&gt;
 &lt;h3&gt;Key Mastering&lt;/h3&gt;
 &lt;p&gt;Let's say you want to store some &lt;a class="reference external" href="https://kobl.one/blog/create-full-ethereum-keypair-and-address/"&gt;Ethereum addresses&lt;/a&gt; that are of interest to you, but should be of interest to noone else.&lt;/p&gt;
 &lt;p&gt;So, let's choose a prefix for &amp;quot;Ethereum addresses.&amp;quot; Let it be &lt;tt class="docutils literal"&gt;etha&lt;/tt&gt;&lt;/p&gt;
 &lt;p&gt;Next, let's pick an identifier that we will remember, that describes the &lt;em&gt;purpose&lt;/em&gt; of the address. In this case it's for &amp;quot;governance of the organization 'Xy Zzy Co.'&amp;quot;. The key then becomes &lt;tt class="docutils literal"&gt;ethagovxyzzyco&lt;/tt&gt;.&lt;/p&gt;
 &lt;p&gt;Last, for a bit of extra fun, let's add some random stuff at the end of the string, like &lt;a class="reference external" href="https://www.quotes.net/mquote/42425"&gt;shilling!shilling?oh,shilling&lt;/a&gt;. Thus, we finally end up with &lt;cite&gt;ethagovxyzshilling!shilling?oh,shilling`&lt;/cite&gt;.&lt;/p&gt;
 &lt;p&gt;What we need to memorize here is:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;whenever referring to an &lt;em&gt;Ethereum address&lt;/em&gt;. we use &lt;tt class="docutils literal"&gt;etha&lt;/tt&gt;&lt;/li&gt;
 &lt;li&gt;whenever referring to &lt;em&gt;governance&lt;/em&gt;. we use &lt;tt class="docutils literal"&gt;gov&lt;/tt&gt;&lt;/li&gt;
 &lt;li&gt;whenever referring to an entity, we use the name of the organization &lt;em&gt;less whitespace and special characters&lt;/em&gt;.&lt;/li&gt;
 &lt;li&gt;the three above are added one after another&lt;/li&gt;
 &lt;li&gt;add the extra fun&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;There will of course always be a pattern to how you create and add similar structures for other categories, actions and entities.&lt;/p&gt;
 &lt;p&gt;But the point here is to show that it doesn't take &lt;em&gt;that much&lt;/em&gt; to have some advantage of &lt;em&gt;plausible deniability&lt;/em&gt;: Even faced with the &lt;a class="reference external" href="https://xkcd.com/538/"&gt;5 Dollar Wrench Problem&lt;/a&gt;, if you are able to hold your ground, there is no feasible way to prove that you &lt;em&gt;don't own something&lt;/em&gt;. Bite the bullet, absorb the pain, deny ownership, and you'll be fine.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="perils-and-pitfalls"&gt;
 &lt;h2&gt;Perils and pitfalls&lt;/h2&gt;
 &lt;p&gt;None of this is in no way safe by itself.&lt;/p&gt;
 &lt;p&gt;Some concerns are named below. Surely there are others, too.&lt;/p&gt;
 &lt;div class="section" id="out-of-sight-but-still-mined"&gt;
 &lt;h3&gt;Out of sight, but still mined&lt;/h3&gt;
 &lt;p&gt;You may find it a pain in the ass to type the passphrase for the key salt every time.&lt;/p&gt;
 &lt;p&gt;An obvious workaround for that is to store the passphrase in a file.&lt;/p&gt;
 &lt;p&gt;But keep in mind that, if you do that, the key obfuscation is going to be less safe than your regular passwords storage.&lt;/p&gt;
 &lt;p&gt;That is to say: If the attacker can read your passwords ciphertext, most likely the attacked can read you obfuscation passphrase, too.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="out-of-mind-out-of-luck"&gt;
 &lt;h3&gt;Out of mind, out of luck&lt;/h3&gt;
 &lt;p&gt;Maybe the most important thing to realize in this solution is that there is no way to recall the original keys from the file names under which the values are stored.&lt;/p&gt;
 &lt;p&gt;Instead, the idea is that you decide on a naming scheme for certain crucial values you want to store, and &lt;em&gt;remember&lt;/em&gt; how to reconstruct them. &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="shell-shill"&gt;
 &lt;h3&gt;Shell shill&lt;/h3&gt;
 &lt;p&gt;The code above has been adapted from a &lt;tt class="docutils literal"&gt;bash&lt;/tt&gt; tool I've already written for the purpose, given the name &lt;a class="reference external" href="https://holbrook.no/src/clortho/file/clortho.sh.html#l10"&gt;Clortho&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Shell script is surely a very poor choice for this kind of thing, but at the time I simply wanted to write shell so that's how it ended up.&lt;/p&gt;
 &lt;p&gt;I probably will rewrite it for another environment at some point.&lt;/p&gt;
 &lt;p&gt;But for now it illustrates the point.&lt;/p&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;I use a different email for each service I sign up to, and for every other context I have to leave my email for something. I highly recommend this practice, aswell as running your own mail server. &lt;a class="reference external" href="https://www.linuxbabe.com/mail-server/setup-basic-postfix-mail-sever-ubuntu"&gt;It really isn't that hard&lt;/a&gt;.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;I'm not going to lie: That kind of thing takes practice. As in, you need to remind yourself from time to time what your scheme is. Perhaps that is an unthinkable proposition in the age of the Smartphone and the illusion of auxiliary instant truths. But I come from a time where I knew all numbers and addresses I needed to know by heart. The more you do it, the more you remember. The less you do it, the more you are dependent on others. What others do you depend on? And so on it goes...&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 </content><category term="Code"></category><category term="crypto"></category><category term="hash"></category><category term="sha512"></category><category term="bash"></category><category term="cli"></category><category term="ccrypt"></category></entry><entry><title>Keeping your gits in a row</title><link href="git-fresh.html" rel="alternate"></link><published>2024-06-18T16:28:00+02:00</published><updated>2024-06-18T16:33:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2024-06-18:git-fresh.html</id><summary type="html">&lt;p class="first last"&gt;Scripts to keep your local git clone fresh, and help you move them around.&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;I believe that if you use a piece of code, you are also responsible for making sure that that code is available in the future.&lt;/p&gt;
 &lt;p&gt;In this spirit, I decided a couple of years ago that I would keep a full clone of all VCS repositories that I use.&lt;/p&gt;
 &lt;div class="section" id="can-t-someone-else-do-it"&gt;
 &lt;h2&gt;Can't someone else do it?&lt;/h2&gt;
 &lt;p&gt;Yeah, yeah, I hear ya.&lt;/p&gt;
 &lt;p&gt;But imagine that one day you cannot reach the code repository anymore.&lt;/p&gt;
 &lt;p&gt;It could be because you are working where internet is scarce or impossible to rely on.&lt;/p&gt;
 &lt;p&gt;It could be that you have to cope with what was in your faraday cage when a giant solar flare happened.&lt;/p&gt;
 &lt;p&gt;It could be that you, or the author of the code, have been cut off by the accelerating &lt;a class="reference external" href="https://torrentfreak.com/the-eu-wants-its-own-dns-resolver-that-can-block-unlawful-traffic-220119/"&gt;weaponization of everything&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Or maybe none of the above happened. But you still understand and appreciate what it means to build a truly decentralized society, where we all participate and contribute, not only consume.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="git-organized"&gt;
 &lt;h2&gt;Git organized&lt;/h2&gt;
 &lt;p&gt;For every &lt;cite&gt;git&lt;/cite&gt; repository that I use, I actually keep a &lt;em&gt;local copy&lt;/em&gt; on my daily device.&lt;/p&gt;
 &lt;p&gt;I also keep a copy on a device at home, &lt;em&gt;and&lt;/em&gt; on a remote device.&lt;/p&gt;
 &lt;p&gt;My thinking is:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;If I lose my laptop, I have two copies&lt;/li&gt;
 &lt;li&gt;If my house burns down, I have two copies&lt;/li&gt;
 &lt;li&gt;If my house burns down &lt;em&gt;with&lt;/em&gt; my laptop inside, I have &lt;em&gt;at least one more copy&lt;/em&gt;.&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;... and so on.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="i-hate-to-move-it-move-it"&gt;
 &lt;h2&gt;I hate to move it, move it&lt;/h2&gt;
 &lt;p&gt;Sometimes we have to, though,.&lt;/p&gt;
 &lt;p&gt;And what can be a real pain is to move heaps of code repositories around. For example if you are moving to a new machine, or want to bootstrap a new copy without having to source the data yourself.&lt;/p&gt;
 &lt;p&gt;To make this easier, I wrote the &lt;a class="reference external" href="https://holbrook.no/src/gitrefresh/log.html"&gt;gitrefresh bash tool&lt;/a&gt; to copy only the minimum of information required to source the data from a remote. &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="freshening-up"&gt;
 &lt;h2&gt;Freshening up&lt;/h2&gt;
 &lt;p&gt;To make sense of what is what in the repository store, I use a simple folder structure.&lt;/p&gt;
 &lt;p&gt;Obviously, when I create copies of the repository store, I would like to keep the same folder structure. So the tool needed to make that possible.&lt;/p&gt;
 &lt;p&gt;Additionally, what's needed are tools to bootstrap a repository group from a list, and a tool to refresh those repositories periodically once they've been bootstrapped.&lt;/p&gt;
 &lt;p&gt;To achieve this, I actually wrote &lt;a class="reference external" href="https://holbrook.no/src/gitrefresh/log.html"&gt;three tools&lt;/a&gt;, as follows:&lt;/p&gt;
 &lt;div class="section" id="gitlist-sh"&gt;
 &lt;h3&gt;&lt;cite&gt;gitlist.sh&lt;/cite&gt;&lt;/h3&gt;
 &lt;p&gt;create a list of &lt;cite&gt;git&lt;/cite&gt; repositories under a filesystem path, with the option of preserving the directory structure.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="gitstart-sh"&gt;
 &lt;h3&gt;&lt;cite&gt;gitstart.sh&lt;/cite&gt;&lt;/h3&gt;
 &lt;p&gt;clone &lt;cite&gt;git&lt;/cite&gt; repositories from a list generated from &lt;code&gt;gitlist.sh&lt;/code&gt;, with or without direcory structure.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="gitrefresh-sh"&gt;
 &lt;h3&gt;&lt;cite&gt;gitrefresh.sh&lt;/cite&gt;&lt;/h3&gt;
 &lt;p&gt;fetch and merge updates from remotes of each repository under a directory.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="behavior"&gt;
 &lt;h2&gt;Behavior&lt;/h2&gt;
 &lt;p&gt;The &lt;code&gt;gitlist.sh&lt;/code&gt; and &lt;code&gt;gitrefresh.sh&lt;/code&gt; tools work more or less the same way.&lt;/p&gt;
 &lt;p&gt;They traverse a directory structure recursively.&lt;/p&gt;
 &lt;p&gt;Every time a valid git repository is found, that repository is processed. Afterwards, the tool will exit to the parent folder. &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/p&gt;
 &lt;div class="section" id="example"&gt;
 &lt;h3&gt;Example&lt;/h3&gt;
 &lt;p&gt;Let's say we have three repositories that we are mirroring locally:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;&lt;code&gt;https://github.com/bitcoin/bips&lt;/code&gt; under &lt;code&gt;btc/bips&lt;/code&gt;&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;https://aur.archlinux.org/libkeccak.git&lt;/code&gt; under &lt;code&gt;os/archlinux/aur/libkeccak&lt;/code&gt;&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git&lt;/code&gt; under &lt;code&gt;linux/linux&lt;/code&gt;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;First we use &lt;code&gt;gitlist.sh&lt;/code&gt; to generate the list of repos to bootstrap &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt;:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$ &lt;/span&gt;gitlist.sh&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tee&lt;span class="w"&gt; &lt;/span&gt;gitlist.txt&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;https://github.com/bitcoin/bips btc/bips
 https://aur.archlinux.org/libkeccak.git os/archlinux/aur/libkeccak`
 git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git  linux/linux`&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Using &lt;code&gt;gitstart.sh&lt;/code&gt; with this list, we can restore this bunch of repositories &lt;em&gt;with&lt;/em&gt;  the same directory structure anywhere else:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/path/to/new/repos/location&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;gitstart.sh&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;gitlist.txt
 &lt;/pre&gt;
 &lt;p&gt;Now, the idea is that from time to time you should get the latest changes from the upstream source.&lt;/p&gt;
 &lt;p&gt;I simply combine &lt;code&gt;gitrefresh.sh&lt;/code&gt; with &lt;code&gt;cron&lt;/code&gt; to do this on the remote, while manually doing the refresh locally once in awhile.&lt;/p&gt;
 &lt;p&gt;Using the tool, all it takes is:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/path/to/new/repos/location&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;gitrefresh.sh&lt;span class="w"&gt; &lt;/span&gt;pull
 &lt;/pre&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Yes. I didn't get beyond &lt;cite&gt;git&lt;/cite&gt; yet. But at least it's a start.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;This, of course, means that the tool will not automatically archive code from &lt;em&gt;submodules&lt;/em&gt;. The submodule construct is a target of both a lot of love and a lot of hate. Personally, I like it. But at the same time it is my opinion that it does not absolve us from &lt;em&gt;knowing&lt;/em&gt; and being &lt;em&gt;mindful&lt;/em&gt; which submodules a repository is using, and thus making sure that we have an independent clone of that repository.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;We add the &lt;code&gt;-p&lt;/code&gt; flag to preserve the directory structure on disk.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 </content><category term="Archiving"></category><category term="git"></category><category term="bash"></category></entry><entry><title>A world clock in your terminal</title><link href="world-clock-terminal.html" rel="alternate"></link><published>2024-06-09T23:22:35+02:00</published><updated>2024-06-09T23:22:35+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2024-06-09:world-clock-terminal.html</id><summary type="html">&lt;p class="first last"&gt;A single character command to display the current time of your favorite places in the world&lt;/p&gt;
 </summary><content type="html">&lt;div class="section" id="zoning-in"&gt;
 &lt;h2&gt;Zoning in&lt;/h2&gt;
 &lt;p&gt;Timezones in Linux isn't a particularly intuitive issue.&lt;/p&gt;
 &lt;p&gt;On my distro, archlinux, time-zone files are located in &lt;tt class="docutils literal"&gt;/usr/share/zoneinfo&lt;/tt&gt;, and there are several different categories of them.&lt;/p&gt;
 &lt;p&gt;I've given up referencing time in the usual three- and four-letter acronyms like &lt;tt class="docutils literal"&gt;CEST&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;EST&lt;/tt&gt; long ago. Not only are they ambiguous in themselves, but not all of them even have records in the zoneinfo tree.&lt;/p&gt;
 &lt;p&gt;The best approach I have found so far is to look at the world continent directories, and look for nearby cities. Your OS alone may not be enough in all cases to figure out the zone for obscure locations, but at least it gets you half the way there.&lt;/p&gt;
 &lt;p&gt;... as in, you ask your friend or associate to name some big cities close to them, and hope you find a match in your filesystem.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="environmental-zones"&gt;
 &lt;h2&gt;Environmental zones&lt;/h2&gt;
 &lt;p&gt;Turns out that the &lt;tt class="docutils literal"&gt;date&lt;/tt&gt; command of GNU/Linux uses the environment variable &lt;tt class="docutils literal"&gt;TZ&lt;/tt&gt;, which lets you define the time-zone you want the output to be translated to.&lt;/p&gt;
 &lt;p&gt;In my case at time of writing:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;America/El_Salvador&lt;span class="w"&gt; &lt;/span&gt;date
 Sun&lt;span class="w"&gt; &lt;/span&gt;Jun&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;05&lt;/span&gt;:00:16&lt;span class="w"&gt; &lt;/span&gt;PM&lt;span class="w"&gt; &lt;/span&gt;CST&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2024&lt;/span&gt;
 
 &lt;span class="c1"&gt;# sigh, see what I mean?&lt;/span&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;CST&lt;span class="w"&gt; &lt;/span&gt;date
 Sun&lt;span class="w"&gt; &lt;/span&gt;Jun&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;11&lt;/span&gt;:04:12&lt;span class="w"&gt; &lt;/span&gt;PM&lt;span class="w"&gt; &lt;/span&gt;CST&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2024&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Ugly, yes. Luckily &lt;tt class="docutils literal"&gt;date&lt;/tt&gt; also lets you format the output.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="times-tabled"&gt;
 &lt;h2&gt;Times, tabled&lt;/h2&gt;
 &lt;p&gt;Now, let's bring those two together.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nv"&gt;z&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;US/Hawaii&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;America/El_Salvador&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;US/Eastern&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Europe/Lisbon&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Europe/Berlin&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Africa/Nairobi&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Asia/Taipei&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
 &lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;z&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
 &lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$z&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+&lt;span class="s1"&gt;&amp;#39;%H:%M:%S (%z)&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="nv"&gt;s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;%-16s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$z&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
 &lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="s2"&gt;\t&lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;This script iterates a list of valid time-zone strings from &lt;tt class="docutils literal"&gt;/usr/share/zoneinfo&lt;/tt&gt;, outputting the zone along with the time and offet in a readable, tabulated format.&lt;/p&gt;
 &lt;p&gt;This results in:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;z
 US/Hawaii&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;-1000&lt;span class="o"&gt;)&lt;/span&gt;
 America/El_Salvador&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;17&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;-0600&lt;span class="o"&gt;)&lt;/span&gt;
 US/Eastern&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="m"&gt;19&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;-0400&lt;span class="o"&gt;)&lt;/span&gt;
 Europe/Lisbon&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="m"&gt;00&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;+0100&lt;span class="o"&gt;)&lt;/span&gt;
 Europe/Berlin&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="m"&gt;01&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;+0200&lt;span class="o"&gt;)&lt;/span&gt;
 Africa/Nairobi&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="m"&gt;02&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;+0300&lt;span class="o"&gt;)&lt;/span&gt;
 Asia/Taipei&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="m"&gt;07&lt;/span&gt;:02:12&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;+0800&lt;span class="o"&gt;)&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;The &lt;tt class="docutils literal"&gt;z&lt;/tt&gt; here is merely an alias mapping added to &lt;tt class="docutils literal"&gt;.bashrc&lt;/tt&gt; &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nb"&gt;alias&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;z&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;bash&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/scripts/timezones.sh&lt;span class="sb"&gt;`&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Voila, now you have a world clock in your terminal at any time.&lt;/p&gt;
 &lt;p&gt;And all it takes is two mere keystrokes.&lt;/p&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Of course, be a bit careful with those aliases. There are even commands in GNU/Linux with only one character, like &lt;tt class="docutils literal"&gt;w&lt;/tt&gt;. If you override them in aliases, your override takes precedence.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Offlining"></category><category term="bash"></category><category term="linux"></category></entry><entry><title>Introducing Wala</title><link href="wala.html" rel="alternate"></link><published>2022-10-05T14:39:00+02:00</published><updated>2022-10-05T14:39:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2022-10-05:wala.html</id><summary type="html">&lt;p class="first last"&gt;A simple HTTP content-addressed storage with cryptographic identity pointers.&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;This little project is heavily inspired by the &amp;quot;Single-Owner Chunk&amp;quot; (previously &amp;quot;Mutable Resource&amp;quot;) concept from the &lt;a class="reference external" href="https://ethswarm.org"&gt;Swarm project&lt;/a&gt; &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Dubbed &lt;a class="reference external" href="https://git.defalsify.org/wala"&gt;wala&lt;/a&gt; &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;, the rust application provides two features:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Store any uploaded data under the SHA256 digest of the content itself. We call this &amp;quot;content addressed&amp;quot; storage.&lt;/li&gt;
 &lt;li&gt;Create a link alias to the content using a keyword and a cryptographic identity.&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;First, let's briefly outline what the first item on the list is.&lt;/p&gt;
 &lt;div class="section" id="curl-up-and-digest"&gt;
 &lt;h2&gt;Curl up and digest&lt;/h2&gt;
 &lt;p&gt;Say an instance of wala is listnening on &lt;code&gt;localhost:8000&lt;/code&gt;.&lt;/p&gt;
 &lt;p&gt;A simple demonstration of the content addressed storage could be:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$&amp;nbsp;&lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;PUT&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000&lt;span class="w"&gt; &lt;/span&gt;--data&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;foo
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae  -&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash.&lt;/p&gt;
 &lt;p&gt;That also means that if the content changes, so does the location.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="get-to-the-pointer"&gt;
 &lt;h2&gt;Get to the pointer&lt;/h2&gt;
 &lt;p&gt;Now let's say you want a permanent link to content that is changing.&lt;/p&gt;
 &lt;p&gt;If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url.&lt;/p&gt;
 &lt;p&gt;But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link.&lt;/p&gt;
 &lt;p&gt;This is usually accomplished with a username and password stored and controlled server-side.&lt;/p&gt;
 &lt;p&gt;But we can also do this by simply using public key cryptography instead.&lt;/p&gt;
 &lt;div class="section" id="claiming-what-is-yours"&gt;
 &lt;h3&gt;Claiming what is yours&lt;/h3&gt;
 &lt;p&gt;Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the &lt;a class="reference external" href="https://sequoia-pgp.org"&gt;&amp;quot;sequoia openpgp&amp;quot;&lt;/a&gt; library.&lt;/p&gt;
 &lt;p&gt;The procedure is straightforward enough:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Sign the content with your pgp key.&lt;/li&gt;
 &lt;li&gt;Add an &lt;cite&gt;Authorization&lt;/cite&gt; header &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt; with your public key and signature.&lt;/li&gt;
 &lt;li&gt;Upload the content with an arbitrary  keyword. &lt;a class="footnote-reference" href="#footnote-4" id="footnote-reference-4"&gt;[4]&lt;/a&gt;&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the &lt;em&gt;key fingerprint&lt;/em&gt; and the &lt;em&gt;keyword&lt;/em&gt;.&lt;/p&gt;
 &lt;p&gt;The content can then be accessed &lt;em&gt;both&lt;/em&gt; using the content address &lt;em&gt;and&lt;/em&gt; the symbolic link.&lt;/p&gt;
 &lt;p&gt;In wala, the symbolic link is referred to as a &lt;em&gt;mutable reference&lt;/em&gt;. We will use this term from now on.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="it-ain-t-pretty-but-it-works"&gt;
 &lt;h3&gt;It ain't pretty, but it works&lt;/h3&gt;
 &lt;p&gt;Let's demonstrate creating such an entry in wala using some commonly available tools. &lt;a class="footnote-reference" href="#footnote-5" id="footnote-reference-5"&gt;[5]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;We will use a fictional gnupg private key with the fingerprint &lt;code&gt;F3FAF668E82EF5124D5187BAEF26F4682343F692&lt;/code&gt; and the keyword &lt;code&gt;foo&lt;/code&gt; to create the mutable reference.&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$&amp;nbsp;&lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bar&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;gpg&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;F3FAF668E82EF5124D5187BAEF26F4682343F692&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;base64&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nv"&gt;pub&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;gpg&lt;span class="w"&gt; &lt;/span&gt;--export&lt;span class="w"&gt; &lt;/span&gt;F3FAF668E82EF5124D5187BAEF26F4682343F692&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;base64&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;w&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;PUT&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/foo&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Authorization: PUBSIG:&lt;/span&gt;&lt;span class="nv"&gt;$pub&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="nv"&gt;$sig&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--data&lt;span class="w"&gt; &lt;/span&gt;bar&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;*   Trying 127.0.0.1:8000...
 &lt;/span&gt;&lt;span class="gp"&gt;  % &lt;/span&gt;Total&lt;span class="w"&gt;    &lt;/span&gt;%&lt;span class="w"&gt; &lt;/span&gt;Received&lt;span class="w"&gt; &lt;/span&gt;%&lt;span class="w"&gt; &lt;/span&gt;Xferd&lt;span class="w"&gt;  &lt;/span&gt;Average&lt;span class="w"&gt; &lt;/span&gt;Speed&lt;span class="w"&gt;   &lt;/span&gt;Time&lt;span class="w"&gt;    &lt;/span&gt;Time&lt;span class="w"&gt;     &lt;/span&gt;Time&lt;span class="w"&gt;  &lt;/span&gt;Current&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;                                 Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to localhost (127.0.0.1) port 8000 (#0)
 &amp;gt; PUT /foo HTTP/1.1
 &amp;gt; Host: localhost:8000
 &amp;gt; User-Agent: curl/7.85.0
 &amp;gt; Accept: */*
 &amp;gt; Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF
 &amp;gt; Content-Length: 3
 &amp;gt; Content-Type: application/x-www-form-urlencoded
 &amp;gt;
 } [3 bytes data]
 * Mark bundle as not supporting multiuse
 &amp;lt; HTTP/1.1 200 OK
 &amp;lt; Server: tiny-http (Rust)
 &amp;lt; Date: Wed,  5 Oct 2022 13:15:37 GMT
 &amp;lt; Content-Type: text/plain; charset=UTF-8
 &amp;lt; Access-Control-Allow-Origin: *
 &amp;lt; Access-Control-Allow-Methods: OPTIONS, PUT, GET
 &amp;lt; Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename
 &amp;lt; Content-Length: 64
 &amp;lt;
 { [64 bytes data]
 100    67  100    64  100     3   3548    166 --:--:-- --:--:-- --:--:--  3722100    67  100    64  100     3   3534    165 --:--:-- --:--:-- --:--:--  3722
 * Connection #0 to host localhost left intact
 32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;bar
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;bar&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;bar&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Now, by changing the data and signature, the data under the mutable resource changes:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$&amp;nbsp;&lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;xyzzy&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;gpg&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;F3FAF668E82EF5124D5187BAEF26F4682343F692&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;base64&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;PUT&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/foo&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Authorization: PUBSIG:&lt;/span&gt;&lt;span class="nv"&gt;$pub&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="nv"&gt;$sig&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--data&lt;span class="w"&gt; &lt;/span&gt;xyzzy&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;xyzzy&lt;/span&gt;
 &lt;/pre&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="building-your-identity"&gt;
 &lt;h2&gt;Building your identity&lt;/h2&gt;
 &lt;p&gt;We can also recreate the symbolic link hash using local tools:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;mktemp&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;foo&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'{ printf &amp;quot;%s&amp;quot;,$1; }'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'s/../\\\\x&amp;amp;/g'&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ne&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nv"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;F3FAF668E82EF5124D5187BAEF26F4682343F692&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'s/../\\\\x&amp;amp;/g'&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ne&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$k&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2  -&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;In other words, the mutable reference is constructed using the following recipe.&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Calculate binary value &lt;code&gt;R&lt;/code&gt; of the SHA256 digest of the keyword (&lt;code&gt;sha256(foo) -&amp;gt; 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE&lt;/code&gt;)&lt;/li&gt;
 &lt;li&gt;Calculate binary value &lt;code&gt;K&lt;/code&gt; of the key fingerprint (&lt;code&gt;0xF3FAF668E82EF5124D5187BAEF26F4682343F692&lt;/code&gt;).&lt;/li&gt;
 &lt;li&gt;Calculate binary value of SHA256 digest of &lt;code&gt;R | K&lt;/code&gt; (&lt;code&gt;sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -&amp;gt; 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2&lt;/code&gt;)&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it.&lt;/p&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Refer to &lt;a class="reference external" href="https://www.ethswarm.org/The-Book-of-Swarm.pdf"&gt;section 4.3 of The Book of Swarm&lt;/a&gt; for a description of &amp;quot;feeds&amp;quot; and &amp;quot;single owner chunks.&amp;quot; Swarm uses signatures to allow propagation of the data in the network. &lt;code&gt;wala&lt;/code&gt; similarly uses signatures to accept update of resources &amp;quot;owned&amp;quot; by the holder of the private keys of an identity.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;At the time of writing &lt;code&gt;wala&lt;/code&gt; is at version &lt;code&gt;0.1.1&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;A custom authorization scheme &lt;code&gt;PUBSIG&lt;/code&gt; has been invented for this purpose. The format is &lt;code&gt;Authorization: PUBSIG pgp:key-fingerprint-in-base64&amp;gt;:&amp;lt;signature-in-base64&amp;gt;&lt;/code&gt;. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-4" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (&lt;code&gt;0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855&lt;/code&gt;)&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-5" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-5"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;wala&lt;/code&gt; comes with a binary tool &lt;code&gt;wala_send&lt;/code&gt; that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Code"></category><category term="wala"></category><category term="pgp"></category><category term="crypto"></category><category term="storage"></category><category term="http"></category><category term="rust"></category></entry><entry><title>A portable book metadata exercise</title><link href="portable-book-metadata.html" rel="alternate"></link><published>2022-10-01T12:40:00+02:00</published><updated>2022-10-01T12:40:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2022-10-01:portable-book-metadata.html</id><summary type="html">&lt;p class="first last"&gt;Structured approach to generate portable metadata files for bibliographies and literature files using cryptographic hash mapping.&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;One of the things I have been working on the last few weeks is a rust application I have dubbed &lt;a class="reference external" href="https://git.defalsify.net/kitab"&gt;kitab&lt;/a&gt; &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;In short, the application makes it easy to extract literary metadata to a separate file structure.&lt;/p&gt;
 &lt;p&gt;The metadata can in turn be applied as &lt;em&gt;extended attributes&lt;/em&gt; recursively on a directory for files that match.&lt;/p&gt;
 &lt;p&gt;The way it's accomplished it simple: The file name of the metadata is the hex representation of the digest of the file. The same digest is used to match files to metadata when applying it back to the file.&lt;/p&gt;
 &lt;p&gt;There are two advantages to this:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;The digest of the media file need not be affected by the metadata, i.e. by embedding metadata in the file itself.&lt;/li&gt;
 &lt;li&gt;You do not need to use the file name to keep record of what a file is.&lt;/li&gt;
 &lt;/ol&gt;
 &lt;div class="section" id="yarr-ye-matey-data"&gt;
 &lt;h2&gt;Yarr, ye matey-data&lt;/h2&gt;
 &lt;p&gt;Let's demonstrate with an example.&lt;/p&gt;
 &lt;p&gt;The fabulous &lt;a class="reference external" href="https://libgen.rs"&gt;Library Genesis&lt;/a&gt; project has made available an endpoint to retrieve &lt;tt class="docutils literal"&gt;bibtex&lt;/tt&gt; entries based on the &lt;tt class="docutils literal"&gt;md5&lt;/tt&gt; hash of the book media file.&lt;/p&gt;
 &lt;p&gt;A version of the &lt;a class="reference external" href="https://libgen.rs/book/index.php?md5=BCD99F1AB4155F2A2A362E5B7938A852"&gt;Bitcoin White Paper&lt;/a&gt;, under the &lt;code&gt;md5&lt;/code&gt; hash &lt;code&gt;bcd99f1ab4155f2a2a362e5b7938a852&lt;/code&gt;, can be found there.&lt;/p&gt;
 &lt;p&gt;If you download this file using a synchronous download link, the browser will provide you with a filename to go with the download.&lt;/p&gt;
 &lt;p&gt;However, if you use the torrent alternative, the filename will be the &lt;tt class="docutils literal"&gt;md5&lt;/tt&gt; hash itself. If you are torrenting a bunch of those files, it quickly becomes a nuisance to distinguish them.&lt;/p&gt;
 &lt;p&gt;And, of course: In either case there is no guarantee the any metadata comes with the file.&lt;/p&gt;
 &lt;div class="section" id="inside-the-book"&gt;
 &lt;h3&gt;Inside the book&lt;/h3&gt;
 &lt;p&gt;Kitab (v0.0.2) is able to read metadata from both a bibtex source and xattr entries on a file, as well as its native &lt;a class="reference external" href="https://www.w3.org/TR/turtle/"&gt;rdf-turtle&lt;/a&gt; format.&lt;/p&gt;
 &lt;p&gt;In kitab's data store, every media file entity in rdf-turtle is keyed with a &lt;a class="reference external" href="https://www.rfc-editor.org/info/rfc8141"&gt;URN&lt;/a&gt; specifying a digest for the file.&lt;/p&gt;
 &lt;p&gt;To see exactly what that looks like, let's download and import the bibtex metadata for the paper &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;:&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 &lt;span class="nv"&gt;bibtex_file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;mktemp&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;kitab_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;https://libgen.rs/book/bibtex.php?md5&lt;span class="o"&gt;=&lt;/span&gt;BCD99F1AB4155F2A2A362E5B7938A852&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$bibtex_file&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;kitab&lt;span class="w"&gt; &lt;/span&gt;--store&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kitab_dir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;import&lt;span class="w"&gt; &lt;/span&gt;--digest&lt;span class="w"&gt; &lt;/span&gt;md5:BCD99F1AB4155F2A2A362E5B7938A852&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$bibtex_file&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kitab_dir&lt;/span&gt;/*
 &lt;/pre&gt;
 &lt;p&gt;The output of the above should be:&lt;/p&gt;
 &lt;pre class="code turtle literal-block"&gt;
 &lt;span class="nv"&gt;&amp;lt;URN:md5:bcd99f1ab4155f2a2a362e5b7938a852&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;&amp;lt;https://purl.org/dc/terms/title&amp;gt;&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Bitcoin: A Peer-to-Peer Electronic Cash System&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;;&lt;/span&gt;
 &lt;span class="nv"&gt;&amp;lt;https://purl.org/dc/terms/creator&amp;gt;&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;Satoshi Nakamoto&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;;&lt;/span&gt;
 &lt;span class="nv"&gt;&amp;lt;https://purl.org/dc/terms/type&amp;gt;&lt;/span&gt; &lt;span class="s"&gt;&amp;quot;book&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Now let's say the media file itself has been downloaded to &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;~/.local/share/transmission&lt;/span&gt;&lt;/tt&gt;. We can apply this metadata as extended attributes.&lt;/p&gt;
 &lt;p&gt;This time we turn on logging to see what's going on:&lt;/p&gt;
 &lt;pre class="code console literal-block"&gt;
 &lt;span class="gp"&gt;$ &lt;/span&gt;&lt;span class="nv"&gt;RUST_LOG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;kitab&lt;span class="w"&gt; &lt;/span&gt;--store&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kitab_dir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--digest&lt;span class="w"&gt; &lt;/span&gt;md5&lt;span class="w"&gt; &lt;/span&gt;~/.local/share/transmission&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;[2022-10-01T11:14:59Z INFO  kitab] have index directory &amp;quot;/tmp/tmp.r0jBm6q4hW&amp;quot;
 [2022-10-01T11:14:59Z INFO  kitab] using digest type md5
 [2022-10-01T11:14:59Z INFO  kitab] apply from path &amp;quot;/home/lash/.local/share/transmission/&amp;quot;
 [2022-10-01T11:14:59Z INFO  kitab] apply DirEntry(&amp;quot;/home/lash/.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852&amp;quot;) -&amp;gt; title &amp;quot;Bitcoin: A Peer-to-Peer Electronic Cash System&amp;quot; author &amp;quot;Satoshi Nakamoto&amp;quot; digest md5:bcd99f1ab4155f2a2a362e5b7938a852
 
 &lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;~/.local/share/transmission&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="w"&gt; &lt;/span&gt;-regextype&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-regex&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.*/[a-f0-9]\{32\}&lt;/span&gt;$&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-exec&lt;span class="w"&gt; &lt;/span&gt;getfattr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="gp"&gt;# &lt;/span&gt;file:&lt;span class="w"&gt; &lt;/span&gt;.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="go"&gt;user.dcterms:creator=&amp;quot;Satoshi Nakamoto&amp;quot;
 user.dcterms:title=&amp;quot;Bitcoin: A Peer-to-Peer Electronic Cash System&amp;quot;
 user.dcterms:type=&amp;quot;book&amp;quot;&lt;/span&gt;
 &lt;/pre&gt;
 &lt;/div&gt;
 &lt;div class="section" id="let-the-right-one-in"&gt;
 &lt;h3&gt;Let the right one in&lt;/h3&gt;
 &lt;p&gt;Conversely, the metadata can be re-imported directly from the extended attributes. And this time, let's store it both under the &lt;tt class="docutils literal"&gt;md5&lt;/tt&gt; and the &lt;tt class="docutils literal"&gt;sha512&lt;/tt&gt; hash:&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;kitab_dir_new&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;kitab&lt;span class="w"&gt; &lt;/span&gt;--store&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kitab_dir_new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;import&lt;span class="w"&gt; &lt;/span&gt;--digest&lt;span class="w"&gt; &lt;/span&gt;md5&lt;span class="w"&gt; &lt;/span&gt;--digest&lt;span class="w"&gt; &lt;/span&gt;sha512&lt;span class="w"&gt; &lt;/span&gt;.local/share/transmission/bcd99f1ab4155f2a2a362e5b7938a852&lt;span class="w"&gt;
 &lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$kitab_dir_new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="w"&gt; &lt;/span&gt;-exec&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;/tmp/tmp.B6j41YMmEM/493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792&lt;span class="w"&gt;
 &lt;/span&gt;&amp;lt;URN:sha512:493f2a720d63156d77187bcd5f0715e4e765a38d616ef47f24e0df817ee6b4f601d47a06ffae10ef1f6ba60bb5d2e99a26318f035f9cd56e30bfe7bcdf64a792&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/title&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Bitcoin: A Peer-to-Peer Electronic Cash System&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/creator&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Satoshi Nakamoto&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/type&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;book&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/MediaType&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;application/epub+zip&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;.&lt;span class="w"&gt;
 &lt;/span&gt;/tmp/tmp.B6j41YMmEM/bcd99f1ab4155f2a2a362e5b7938a852&lt;span class="w"&gt;
 &lt;/span&gt;&amp;lt;URN:md5:bcd99f1ab4155f2a2a362e5b7938a852&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/title&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Bitcoin: A Peer-to-Peer Electronic Cash System&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/creator&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Satoshi Nakamoto&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/type&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;book&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;lt;https://purl.org/dc/terms/MediaType&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;application/epub+zip&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;.
 &lt;/pre&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="level-up"&gt;
 &lt;h2&gt;Level up&lt;/h2&gt;
 &lt;p&gt;Finally, a bash script &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt; example that lets you retrieve and apply metadata for a batch of files found in the directory given as the &lt;em&gt;first positional arg&lt;/em&gt;.&lt;/p&gt;
 &lt;p&gt;This script even renames the files according to the metadata applied.&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 &lt;span class="ln"&gt; 0 &lt;/span&gt;&lt;span class="c1"&gt;# NOTE! this will only work if your fs supports xattr.
 &lt;/span&gt;&lt;span class="ln"&gt; 1 &lt;/span&gt;&lt;span class="c1"&gt;# That's why we cannot use tmpfs (mktemp) here; tmpfs does not support xattr.
 &lt;/span&gt;&lt;span class="ln"&gt; 2 &lt;/span&gt;&lt;span class="c1"&gt;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 3 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="c1"&gt;# directory to copy media files to
 &lt;/span&gt;&lt;span class="ln"&gt; 4 &lt;/span&gt;&lt;span class="c1"&gt;&lt;/span&gt;&lt;span class="nv"&gt;outdir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;./&lt;span class="k"&gt;$(&lt;/span&gt;uuidgen&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 5 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-vp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 6 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 7 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="c1"&gt;# Input dir is the first positional arg.
 &lt;/span&gt;&lt;span class="ln"&gt; 8 &lt;/span&gt;&lt;span class="c1"&gt;&lt;/span&gt;&lt;span class="nv"&gt;indir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 9 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;10 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;$'\n'&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;11 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;12 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="c1"&gt;# Retrieve metadata for each file and import it into the kitab store.
 &lt;/span&gt;&lt;span class="ln"&gt;13 &lt;/span&gt;&lt;span class="c1"&gt;# Also copy the media file to the separate output directory.
 &lt;/span&gt;&lt;span class="ln"&gt;14 &lt;/span&gt;&lt;span class="c1"&gt;&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$indir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;15 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;sum&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;md5sum&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'{print $1;}'&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;16 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;downloading metadata for &lt;/span&gt;&lt;span class="nv"&gt;$indir&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;17 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;srct&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;18 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;https://libgen.rs/book/bibtex.php?md5&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$sum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$srct&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;19 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;dstt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;20 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;xmllint&lt;span class="w"&gt; &lt;/span&gt;--html&lt;span class="w"&gt; &lt;/span&gt;--xpath&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'string(/html/body/textarea[&amp;#64;id=&amp;quot;bibtext&amp;quot;])'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$srct&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$dstt&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;21 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;kitab&lt;span class="w"&gt; &lt;/span&gt;import&lt;span class="w"&gt; &lt;/span&gt;--digest&lt;span class="w"&gt; &lt;/span&gt;md5:&lt;span class="nv"&gt;$sum&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$dstt&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;22 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;/&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;23 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;24 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;25 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="c1"&gt;# Apply metadata imported from bibtex as xattr for the media files.
 &lt;/span&gt;&lt;span class="ln"&gt;26 &lt;/span&gt;&lt;span class="c1"&gt;&lt;/span&gt;&lt;span class="nv"&gt;RUST_LOG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;kitab&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--digest&lt;span class="w"&gt; &lt;/span&gt;md5&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;/&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;27 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;28 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="c1"&gt;# Rename the files according to the metadata title and media type.
 &lt;/span&gt;&lt;span class="ln"&gt;29 &lt;/span&gt;&lt;span class="c1"&gt;&lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;ls&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;30 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;getfattr&lt;span class="w"&gt; &lt;/span&gt;--only-values&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;user.dcterms:title&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;/&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;31 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;32 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;f_typ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;file&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;--mime-type&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;/&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;33 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;34 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$f_typ&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;35 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;application/pdf&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;36 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.pdf&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;37 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;38 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;application/epub+zip&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;39 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.epub&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;40 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;41 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;application/x-mobipocket-ebook&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;42 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.mobi&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;43 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;44 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;text/plain&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;45 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.txt&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;46 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;47 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;text/html&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;48 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.html&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;49 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;50 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;51 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;unhandled&lt;span class="w"&gt; &lt;/span&gt;mime&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f_typ&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;52 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;53 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;esac&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;54 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;mv&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;/&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$outdir&lt;/span&gt;/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;title&lt;/span&gt;&lt;span class="si"&gt;}${&lt;/span&gt;&lt;span class="nv"&gt;f_ext&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;55 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;This last example will result in:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;A media file named &lt;tt class="docutils literal"&gt;$outdir/Bitcoin: A &lt;span class="pre"&gt;Peer-to-Peer&lt;/span&gt; Electronic Cash System.epub&lt;/tt&gt;&lt;/li&gt;
 &lt;li&gt;... with metadata applied as extended attributes&lt;/li&gt;
 &lt;li&gt;An rdf-turtle metadata entry in &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;~/.local/share/kitab/idx/bcd99f1ab4155f2a2a362e5b7938a852&lt;/span&gt;&lt;/tt&gt;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;The relevant documentation for &lt;tt class="docutils literal"&gt;kitab&lt;/tt&gt; at the time of writing is &lt;a class="reference external" href="https://defalsify.org/doc/crates/kitab/0.0.2/kitab/"&gt;here&lt;/a&gt;. To build kitab, simply &lt;em&gt;clone&lt;/em&gt; the repository and build with &lt;code&gt;cargo build --all-features&lt;/code&gt;.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;The &lt;code&gt;kitab&lt;/code&gt; command in the script assumes you have built the &lt;em&gt;kitab binary&lt;/em&gt; and made it available in your path.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;the script uses &lt;code&gt;xmllint&lt;/code&gt; which on archlinux is provided by the &lt;tt class="docutils literal"&gt;libxml2&lt;/tt&gt; package.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Archiving"></category><category term="hash"></category><category term="kitab"></category><category term="literature"></category><category term="metadata"></category><category term="dublincore"></category><category term="libgen"></category></entry><entry><title>Combining duplicity and rsync</title><link href="backup-rsync-duplicity.html" rel="alternate"></link><published>2022-01-15T16:57:00+01:00</published><updated>2022-01-15T16:57:00+01:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2022-01-15:backup-rsync-duplicity.html</id><summary type="html">&lt;p class="first last"&gt;An exercise in combining plain and encrypted backups on local and remote hosts&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;There are two awesome, weathered tools out there that are all you really need for your personal backups. &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt; One is the &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync cli&lt;/a&gt;, the other is &lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;duplicity&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;The former should need no introduction.&lt;/p&gt;
 &lt;p&gt;The latter operates more like tar. But it still works over ssh like rsync. In fact, it's based on &lt;a class="reference external" href="http://librsync.sourcefrog.net/"&gt;librsync&lt;/a&gt; which implements the &lt;a class="reference external" href="https://rsync.samba.org/tech_report/"&gt;rsync protocol&lt;/a&gt;. The special sauce however is, of course, &lt;em&gt;encryption&lt;/em&gt;.&lt;/p&gt;
 &lt;div class="section" id="backup-categories"&gt;
 &lt;h2&gt;Backup categories&lt;/h2&gt;
 &lt;p&gt;Let's for the sake of argument say that our personal backups can be divided in three categories:&lt;/p&gt;
 &lt;div class="section" id="stuff-that-can-be-public"&gt;
 &lt;h3&gt;Stuff that can be public&lt;/h3&gt;
 &lt;p&gt;Code snippets, git repositories, public data store states (e.g. blockchain ledgers), copies of OS packages and any other assets assets without redistribution issues.&lt;/p&gt;
 &lt;p&gt;For this we will use &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync&lt;/a&gt;.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="sensitive-stuff"&gt;
 &lt;h3&gt;Sensitive stuff&lt;/h3&gt;
 &lt;p&gt;Passwords, keys, contacts, calendars, contracts, invoices, task lists, databases, system configurations, application data.&lt;/p&gt;
 &lt;p&gt;For this we will use &lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;duplicity&lt;/a&gt;.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="secret-stuff"&gt;
 &lt;h3&gt;Secret stuff&lt;/h3&gt;
 &lt;p&gt;Long-lived keys, password- and volume decryption keys, cryptocurrency keys and meta-information about the backups themselves.&lt;/p&gt;
 &lt;p&gt;This will not be addressed now.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="why-not-just-one-or-the-other"&gt;
 &lt;h2&gt;Why not just one or the other?&lt;/h2&gt;
 &lt;p&gt;&lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;Duplicity&lt;/a&gt; stores everything in an archive file format. That means that you must first authenticate, decrypt and unpack the archive in order to even browse the files inside.&lt;/p&gt;
 &lt;p&gt;If there is no reason to keep the files from prying eyes, then it's much more practical to be able to browse the files where they lie, with the regular filesystem tools. In such a case, &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync&lt;/a&gt; will scratch your itch.&lt;/p&gt;
 &lt;p&gt;For the &lt;strong&gt;sensitive&lt;/strong&gt; and &lt;strong&gt;secret stuff&lt;/strong&gt;, there would be no real need to use &lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;duplicity&lt;/a&gt; if you were only operating on your local host. You'd just use an encrypted volume &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt; and &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync&lt;/a&gt; everything in there.&lt;/p&gt;
 &lt;p&gt;But half the point here is to keep remote copies aswell as your local ones. You know, in case of fire, hardware-eating locust swarms or some totalitarian minions nabbing all your electronics. Unless &amp;quot;remote&amp;quot; here means some box hidden in some moated leisure castle of yours, you'll want to encrypt everything &lt;em&gt;before&lt;/em&gt; you ship it off. And that's where &lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;duplicity&lt;/a&gt; comes in.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="vive-la-difference"&gt;
 &lt;h2&gt;Vive la difference&lt;/h2&gt;
 &lt;p&gt;Of course, it would be too much to hope for that &lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;duplicity&lt;/a&gt; and &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync cli&lt;/a&gt; have aligned the ways they parse their invocation parameters.&lt;/p&gt;
 &lt;p&gt;Here are some examples &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt; of how they do &lt;em&gt;not&lt;/em&gt; match:&lt;/p&gt;
 &lt;div class="section" id="local-to-local"&gt;
 &lt;h3&gt;local to local&lt;/h3&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;rsync&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;src/&lt;span class="w"&gt; &lt;/span&gt;/path/to/dst/
 
 $&lt;span class="w"&gt; &lt;/span&gt;duplicity&lt;span class="w"&gt; &lt;/span&gt;src/&lt;span class="w"&gt; &lt;/span&gt;file:///path/to/dst/
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="local-to-remote-relative-path"&gt;
 &lt;h3&gt;local to remote, relative path&lt;/h3&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;rsync&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;src/&lt;span class="w"&gt; &lt;/span&gt;user@remotehost:path/to/dst/
 
 $&lt;span class="w"&gt; &lt;/span&gt;duplicity&lt;span class="w"&gt; &lt;/span&gt;src/&lt;span class="w"&gt; &lt;/span&gt;scp://user@remotehost/path/to/dst
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="toggle-dotfiles-from-current-path"&gt;
 &lt;h3&gt;toggle dotfiles from current path&lt;/h3&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# include only .foo/foo.txt given the current structure:&lt;/span&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;tree&lt;span class="w"&gt; &lt;/span&gt;src/&lt;span class="w"&gt; &lt;/span&gt;-a
 src/
 ├──&lt;span class="w"&gt; &lt;/span&gt;.bar
 ├──&lt;span class="w"&gt; &lt;/span&gt;baz
 └──&lt;span class="w"&gt; &lt;/span&gt;.foo
 &lt;span class="w"&gt;    &lt;/span&gt;└──&lt;span class="w"&gt; &lt;/span&gt;foo.txt
 
 $&lt;span class="w"&gt; &lt;/span&gt;rsync&lt;span class="w"&gt; &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.b*&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--include&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.*/***&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;./&lt;span class="w"&gt; &lt;/span&gt;../dst/
 
 $&lt;span class="w"&gt; &lt;/span&gt;duplicity&lt;span class="w"&gt; &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./.b*&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--include&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;./.*/***&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;./&lt;span class="w"&gt; &lt;/span&gt;file:///home/lash/tmp/dst/
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="logging"&gt;
 &lt;h3&gt;logging&lt;/h3&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# spill the beans&lt;/span&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;rsync&lt;span class="w"&gt; &lt;/span&gt;-vv&lt;span class="w"&gt; &lt;/span&gt;...
 
 $&lt;span class="w"&gt; &lt;/span&gt;duplicity&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;debug
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="batchin"&gt;
 &lt;h2&gt;Batchin'&lt;/h2&gt;
 &lt;p&gt;Since you will want to select up front which tool to use for which sensititivy category, you'll be writing the includes and excludes specifically for the tool anyway.&lt;/p&gt;
 &lt;p&gt;So the only real issue with the above is the way remote host is specified.&lt;/p&gt;
 &lt;p&gt;Let's say we choose to stick to the &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync cli&lt;/a&gt; host format. That means we need to make the following translations:&lt;/p&gt;
 &lt;table border="1" class="docutils"&gt;
 &lt;colgroup&gt;
 &lt;col width="50%" /&gt;
 &lt;col width="50%" /&gt;
 &lt;/colgroup&gt;
 &lt;thead valign="bottom"&gt;
 &lt;tr&gt;&lt;th class="head"&gt;rsync&lt;/th&gt;
 &lt;th class="head"&gt;duplicity&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td&gt;&lt;tt class="docutils literal"&gt;foo/bar&lt;/tt&gt;&lt;/td&gt;
 &lt;td&gt;&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;file://foo/bar&lt;/span&gt;&lt;/tt&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;tt class="docutils literal"&gt;/foo/bar&lt;/tt&gt;&lt;/td&gt;
 &lt;td&gt;&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;file:///foo/bar&lt;/span&gt;&lt;/tt&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;tt class="docutils literal"&gt;user&amp;#64;host:foo/bar&lt;/tt&gt;&lt;/td&gt;
 &lt;td&gt;&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;scp://user&amp;#64;host/foo/bar&lt;/span&gt;&lt;/tt&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;user&amp;#64;host:/foo/bar&lt;/span&gt;&lt;/tt&gt;&lt;/td&gt;
 &lt;td&gt;&lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;scp://user&amp;#64;host//foo/bar&lt;/span&gt;&lt;/tt&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;p&gt;Expressed in &lt;tt class="docutils literal"&gt;bash&lt;/tt&gt; that could look like this:&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 to_duplicity_remote&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# remote host is defined in rsync format
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# ... and we will only support scp
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# $remote_base is the path we want to parse
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;remote_base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# substring up until the first slash
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;s_firstslash&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;remote_base&lt;/span&gt;&lt;span class="p"&gt;%%/*&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# substring up until the first colon
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;s_firstcolon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;remote_base&lt;/span&gt;&lt;span class="p"&gt;%%:*&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# string index of the first slash
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;i_firstslash&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;s_firstslash&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# string index of the first colon
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;i_firstcolon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;s_firstcolon&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# if colon is before first slash that most likely means we have a remote host
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# (Exception is if first directory of path happens to have &amp;quot;:&amp;quot; in it. Seriously, don't use &amp;quot;:&amp;quot; in filenames)
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$i_firstcolon&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$i_firstcolon&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-lt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$i_firstslash&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
 
                         &lt;/span&gt;&lt;span class="c1"&gt;# pexpect addition due to lack of implicit private key fetch, without pexpect works only with key wo pwd
 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="c1"&gt;# https://serverfault.com/questions/982591/duplicity-backup-fails-private-key-file-is-encrypted
 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pexpect+scp://&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 
                         &lt;/span&gt;&lt;span class="c1"&gt;# trim url so that colon after hostname is removed
 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="c1"&gt;# (no support here for setting an alternate port number)
 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="si"&gt;}${&lt;/span&gt;&lt;span class="nv"&gt;remote_base&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nv"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nv"&gt;$i_firstcolon&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;remote_base&lt;/span&gt;&lt;span class="p"&gt;:((&lt;/span&gt;&lt;span class="nv"&gt;$i_firstcolon&lt;/span&gt;&lt;span class="p"&gt;+1))&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
                         &lt;/span&gt;&lt;span class="c1"&gt;# indicate that we have a remote host
 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nv"&gt;remote&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="c1"&gt;# If it's not a remote host, treat it as a file
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$remote_duplicity_base&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="nv"&gt;remote_duplicity_base&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;file://&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;remote_base&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BAK_TEST&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;/foo/bar&lt;span class="w"&gt; &lt;/span&gt;foo/bar&lt;span class="w"&gt; &lt;/span&gt;localhost:foo/bar&lt;span class="w"&gt; &lt;/span&gt;localhost:/foo/bar&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;res&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;file:///foo/bar&lt;span class="w"&gt; &lt;/span&gt;file://foo/bar&lt;span class="w"&gt; &lt;/span&gt;pexpect+scp://localhost/foo/bar&lt;span class="w"&gt; &lt;/span&gt;pexpect+scp://localhost//foo/bar&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;case_src&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="p"&gt;[&amp;#64;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="nv"&gt;case_res&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;res&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;to_duplicity_remote&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$remote_duplicity_base&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$case_res&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;expected &lt;/span&gt;&lt;span class="nv"&gt;$case_res&lt;/span&gt;&lt;span class="s2"&gt; got &lt;/span&gt;&lt;span class="nv"&gt;$remote_duplicity_base&lt;/span&gt;&lt;span class="s2"&gt; from &lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;elif&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$remote_base&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="s2"&gt; got mangled into &lt;/span&gt;&lt;span class="nv"&gt;$remote_base&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Let's behave and test our code:&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BAK_TEST&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;/foo/bar&lt;span class="w"&gt; &lt;/span&gt;foo/bar&lt;span class="w"&gt; &lt;/span&gt;localhost:foo/bar&lt;span class="w"&gt; &lt;/span&gt;localhost:/foo/bar&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;res&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;file:///foo/bar&lt;span class="w"&gt; &lt;/span&gt;file://foo/bar&lt;span class="w"&gt; &lt;/span&gt;pexpect+scp://localhost/foo/bar&lt;span class="w"&gt; &lt;/span&gt;pexpect+scp://localhost//foo/bar&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;case_src&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;src&lt;/span&gt;&lt;span class="p"&gt;[&amp;#64;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="nv"&gt;case_res&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;res&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;to_duplicity_remote&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$remote_duplicity_base&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$case_res&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;expected &lt;/span&gt;&lt;span class="nv"&gt;$case_res&lt;/span&gt;&lt;span class="s2"&gt; got &lt;/span&gt;&lt;span class="nv"&gt;$remote_duplicity_base&lt;/span&gt;&lt;span class="s2"&gt; from &lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;elif&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$remote_base&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$case_src&lt;/span&gt;&lt;span class="s2"&gt; got mangled into &lt;/span&gt;&lt;span class="nv"&gt;$remote_base&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
 &lt;/pre&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# 0 == good!&lt;/span&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;BAK_TEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bash&lt;span class="w"&gt; &lt;/span&gt;remote.sh&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;
 &lt;span class="m"&gt;0&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Now we can use the &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync cli&lt;/a&gt; path input, and use that same input to a batch of single backup steps, each which may use &lt;a class="reference external" href="https://rsync.samba.org/"&gt;rsync cli&lt;/a&gt; or &lt;a class="reference external" href="https://duplicity.gitlab.io/duplicity-web/"&gt;duplicity&lt;/a&gt;&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;to_duplicity_remote&lt;span class="w"&gt; &lt;/span&gt;localhost:/foo/bar
 
 rsync&lt;span class="w"&gt; &lt;/span&gt;-avzP&lt;span class="w"&gt; &lt;/span&gt;pub/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$remote_base&lt;/span&gt;:src/
 
 duplicity&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;secret/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$remote_duplicity_base&lt;/span&gt;:secret/
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="see-also"&gt;
 &lt;h2&gt;See also&lt;/h2&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;&lt;a class="reference external" href="https://git.defalsify.org/rsync-duplicity-backups"&gt;https://git.defalsify.org/rsync-duplicity-backups&lt;/a&gt;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Ok, I know, I assuming that you are using &lt;tt class="docutils literal"&gt;git&lt;/tt&gt; in daily life, too.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Provided, of course, that it's an encrypted volume that you don't keep unlocked all the time.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Duplicity needs at a minimum a password for symmetric encryption, and will prompt for it unless it's set in the environment. Simply &lt;tt class="docutils literal"&gt;export PASSPHRASE=test&lt;/tt&gt; for these examples to relieve you of the annoyance.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Archiving"></category><category term="backup"></category><category term="rsync"></category><category term="duplicity"></category><category term="bash"></category></entry><entry><title>Homemade internet state monitor</title><link href="internet-up-monitor.html" rel="alternate"></link><published>2021-07-01T13:59:00+02:00</published><updated>2021-07-01T13:59:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-07-01:internet-up-monitor.html</id><summary type="html">&lt;p class="first last"&gt;A small script to detect internet status&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;I use &lt;a class="reference external" href="https://i3wm.org"&gt;i3wm&lt;/a&gt;. I use it because it has a very low resource footprint, it has great documentation, and it's simple to configure.&lt;/p&gt;
 &lt;p&gt;Complementing &lt;em&gt;i3wm&lt;/em&gt; is the &lt;em&gt;i3status&lt;/em&gt; status provider to the &lt;em&gt;i3bar&lt;/em&gt;. It provides a good suite of preset components for things like temperatur measurement, disk capacity. It also provides a couple of generic ones, namely &amp;quot;output the contents of a file&amp;quot; or &amp;quot;yes if file exists, no if not.&amp;quot;&lt;/p&gt;
 &lt;div class="section" id="it-s-not-you-it-s-me"&gt;
 &lt;h2&gt;It's not you, it's me?&lt;/h2&gt;
 &lt;div class="admonition warning"&gt;
 &lt;p class="first admonition-title"&gt;Warning&lt;/p&gt;
 &lt;p class="last"&gt;&lt;strong&gt;Always use a VPN! Always!!&lt;/strong&gt;&lt;/p&gt;
 &lt;/div&gt;
 &lt;p&gt;Now, I &lt;em&gt;always&lt;/em&gt; use VPN &lt;em&gt;no matter what&lt;/em&gt;, and I configure network manually. That is, I do not use a network manager. For this reason, when network applications start misbehaving, it's sometimes hard to tell whether it is because the host is gone, the local network is flaky, the internet route is flaky, or simply that the connection is gone for good.&lt;/p&gt;
 &lt;p&gt;Evidently, it would be practical to have &lt;strong&gt;an item in the status bar telling me whether I have access to internet right now or not&lt;/strong&gt;.&lt;/p&gt;
 &lt;p&gt;The &amp;quot;yes if file exists, no if not&amp;quot; option seems to be the right one to go for. Basically, we need to stick an empty file in a runtime file store whenever we detect that internet is available.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="ping-rinse-repeat"&gt;
 &lt;h2&gt;Ping, rinse, repeat&lt;/h2&gt;
 &lt;p&gt;There's not much trickery involved here. The check is a ping to a remote location every now and then.&lt;/p&gt;
 &lt;p&gt;Some caveats to cover to grant it some minimum of intelligence:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;We need a few alternative destinations for the ping, that can take over if others happen to be down or take too long to repond.&lt;/li&gt;
 &lt;li&gt;To make the script useful for general purpose use, we'd also want to be able to supply a custom list of hosts to ping.&lt;/li&gt;
 &lt;li&gt;We also want to make sure that when the script is interrupted, it deletes the internet indicator file. That way, when it stops running, the internet indicator will be frozen in the &amp;quot;no&amp;quot; state.&lt;/li&gt;
 &lt;li&gt;Lastly, it can be useful to log to syslog every time the state of the internet connection (or the script itself) changes.&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;That can translate to a shell script like this;&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 &lt;span class="ch"&gt;#!/bin/bash
 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="c1"&gt;# this is our runtime file, if - which exists - tells us internet is up
 &lt;/span&gt;&lt;span class="nv"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/run/user/&lt;/span&gt;&lt;span class="nv"&gt;$UID&lt;/span&gt;&lt;span class="s2"&gt;/probe_up&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;up&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;iup up&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;touch&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;down&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;iup down&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;argh&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;iup die&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="nv"&gt;IUP_DELAY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IUP_DELAY&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$IUP_DELAY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;IUP_DELAY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;debug&lt;span class="w"&gt; &lt;/span&gt;iup&lt;span class="w"&gt; &lt;/span&gt;started&lt;span class="w"&gt; &lt;/span&gt;with&lt;span class="w"&gt; &lt;/span&gt;delay&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$IUP_DELAY&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="nv"&gt;hosts&lt;/span&gt;&lt;span class="o"&gt;=()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;h&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;hosts&lt;/span&gt;&lt;span class="o"&gt;+=(&lt;/span&gt;&lt;span class="nv"&gt;$h&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/.config/iup&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/null&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;[&amp;#64;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-eq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;hosts&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="m"&gt;8&lt;/span&gt;.8.8.8&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;151&lt;/span&gt;.101.193.67&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# google nameserver and cnn.com
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;warn&lt;span class="w"&gt; &lt;/span&gt;missing&lt;span class="w"&gt; &lt;/span&gt;hosts&lt;span class="w"&gt; &lt;/span&gt;input&lt;span class="w"&gt; &lt;/span&gt;file,&lt;span class="w"&gt; &lt;/span&gt;using&lt;span class="w"&gt; &lt;/span&gt;evil&lt;span class="w"&gt; &lt;/span&gt;default:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;[&amp;#64;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="nb"&gt;trap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;argh&lt;span class="w"&gt; &lt;/span&gt;SIGINT&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nb"&gt;trap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;argh&lt;span class="w"&gt; &lt;/span&gt;SIGTERM&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nb"&gt;trap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;argh&lt;span class="w"&gt; &lt;/span&gt;SIGQUIT&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;true&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;isup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;h&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;[&amp;#64;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;ping&lt;span class="w"&gt; &lt;/span&gt;-c1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$h&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-w1&lt;span class="w"&gt; &lt;/span&gt;-q&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/null&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-eq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'0'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;up&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nv"&gt;isup&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="c1"&gt;#logger -p debug &amp;quot;ping success $h&amp;quot;
 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$isup&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;down&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;sleep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$IUP_DELAY&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Since we'll want this to run when network interfaces allegedly are up, it makes sense to link it to systemd and the network target:&lt;/p&gt;
 &lt;pre class="code ini literal-block"&gt;
 &lt;span class="k"&gt;[Unit]&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Internet connection poller&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/iup.sh&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;[Install]&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Stick this in &lt;code&gt;~/.config/systemd/user/iup.service&lt;/code&gt; and enable and start the service:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;iup.service
 $&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;iup.service
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="checking-what-status-our-status-is-in"&gt;
 &lt;h2&gt;Checking what status our status is in&lt;/h2&gt;
 &lt;p&gt;Now, in the i3status configuration, assuming that your system uid is 1000 &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;, add:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;order += &amp;quot;run_watch INET&amp;quot;
 
 kj
 
 run_watch INET {
         pidfile = &amp;quot;/run/user/1000/probe_up&amp;quot;
 }
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;To reload the config on the fly, press &lt;code&gt;mod+r&lt;/code&gt; (that's &lt;code&gt;ctrl+r&lt;/code&gt; on mine). The result should be a new item in your bar showing &lt;code&gt;INET: yes&lt;/code&gt; (most likely yes, anyway, since you're currently reading this).&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="safety-first-eh-second"&gt;
 &lt;h2&gt;Safety first, eh ... second&lt;/h2&gt;
 &lt;div class="admonition warning"&gt;
 &lt;p class="first admonition-title"&gt;Warning&lt;/p&gt;
 &lt;p class="last"&gt;&lt;strong&gt;Did I mention that you should always use a VPN?&lt;/strong&gt;&lt;/p&gt;
 &lt;/div&gt;
 &lt;p&gt;The same &lt;code&gt;run_watch&lt;/code&gt; trick can be used for VPN.&lt;/p&gt;
 &lt;p&gt;I use &lt;code&gt;openvpn&lt;/code&gt;, and it defines a flag &lt;code&gt;--writepid&lt;/code&gt;. If you pass a file path to this parameter, either through the &lt;code&gt;writepid&lt;/code&gt; config directiry or on the &lt;code&gt;--writepid&lt;/code&gt; flag on the command line, it will write the openvpn pid to that file, and similarly remove it when the openvpn service ends.&lt;/p&gt;
 &lt;p&gt;However, this raises another issue. Namely the fact that a location is needed for openvpn to write the file to, for which it also has access.&lt;/p&gt;
 &lt;p&gt;If we want to use the &lt;code&gt;/run&lt;/code&gt; directory again, now we also has to make sure that this directory exists.&lt;/p&gt;
 &lt;div class="section" id="got-the-runs"&gt;
 &lt;h3&gt;Got the runs?&lt;/h3&gt;
 &lt;p&gt;This is exactly what &lt;code&gt;systemd-tmpfiles&lt;/code&gt; is for. You can add files to &lt;code&gt;/etc/tmpfiles.d&lt;/code&gt; which describe what kind of temporary files or directories to add &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Create a file &lt;code&gt;/etc/tmpfiles.d/openvpn.conf&lt;/code&gt; and add a single line to it:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;d       /run/openvpn 0755    openvpn openvpn
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;This will ensure that the folder is created at startup.&lt;/p&gt;
 &lt;p&gt;Now, to create one right away for the current session, run &lt;code&gt;systemd-tmpfiles --create&lt;/code&gt;.&lt;/p&gt;
 &lt;p&gt;Now we have everything we need to add the VPN status to &lt;em&gt;i3status&lt;/em&gt; aswell. Provided that we chose &lt;code&gt;/run/openvpn/openvpn.pid&lt;/code&gt; as our argument to &lt;code&gt;openvpn --writepid&lt;/code&gt;, the setup is more or less the same as before:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;order += &amp;quot;run_watch VPN&amp;quot;
 
 kj
 
 run_watch VPN {
         pidfile = &amp;quot;/run/openvpn/openvpn.pid&amp;quot;
 }
 &lt;/pre&gt;&lt;/div&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;The &lt;code&gt;/run/user/$UID&lt;/code&gt; folder will be created on system startup, and will be writable by that user. It provides a tmpfs storage location for processes that run in early stages of boot, before the entire filesystem tree (which may include &lt;code&gt;/var/run&lt;/code&gt; because &lt;code&gt;/var&lt;/code&gt; may be on a separate partition) has been mounted. More in that here: &lt;a class="reference external" href="https://lwn.net/Articles/436012/"&gt;https://lwn.net/Articles/436012/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;This service has an impressive amount of flexibility, which you can inspect yourself by visiting the &lt;code&gt;tmpfiles.d&lt;/code&gt; (and &lt;code&gt;systemd-tmpfiles&lt;/code&gt;) manpage.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 </content><category term="Hygiene"></category><category term="bash"></category><category term="systemd"></category><category term="network"></category><category term="i3wm"></category><category term="tmpfs"></category><category term="openvpn"></category></entry><entry><title>Making sense of Ethereum log bloom filters</title><link href="eth-log-bloom.html" rel="alternate"></link><published>2021-06-27T11:17:00+02:00</published><updated>2021-06-27T15:56:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-06-27:eth-log-bloom.html</id><summary type="html">&lt;p class="first last"&gt;How to generate values to match the log bloom filters in Ethereum&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;Ethereum blocks and transaction receipts both carry a bloom filter. They provide a shortcut to check whether a particular event of a particular contract was triggered. The filter in the transaction indexes all events that occurred in the transaction. The filter in the block carries all the filter bits of the individual transactions within the block.&lt;/p&gt;
 &lt;p&gt;Somewhat surprisingly, there are practically no friendly descriptions out there that tell you how to generate filters to match them with. In fact, the &lt;a class="reference external" href="https://holbrook.no/doc/ethereum/yellowpaper.pdf"&gt;Ethereum Yellow Paper&lt;/a&gt; was the only source I found.&lt;/p&gt;
 &lt;p&gt;&lt;strong&gt;So let's make a friendly description&lt;/strong&gt;&lt;/p&gt;
 &lt;p&gt;Before we get to the friendliness, though, an ever so slight dose of brain melt is due:&lt;/p&gt;
 &lt;div class="section" id="the-formalities"&gt;
 &lt;h2&gt;The formalities&lt;/h2&gt;
 &lt;p&gt;The formal definition of the filter calculation in the &lt;a class="reference external" href="https://holbrook.no/doc/ethereum/yellowpaper.pdf"&gt;Ethereum Yellow Paper section 4.3.1&lt;/a&gt; is (here cited in painfully inadequate math formatting):&lt;/p&gt;
 &lt;blockquote&gt;
 &lt;div class="formula"&gt;
 &lt;i&gt;M&lt;/i&gt;(&lt;i&gt;O&lt;/i&gt;) ≡ &lt;span class="limits"&gt;&lt;span class="limit"&gt;&lt;span class="bigoperator"&gt;⋁&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;i&gt;x&lt;/i&gt; ∈ {&lt;i&gt;Oa&lt;/i&gt;}∪&lt;i&gt;Ot&lt;/i&gt;(&lt;i&gt;M&lt;/i&gt;&lt;sub&gt;3 : 2048&lt;/sub&gt;(&lt;i&gt;x&lt;/i&gt;))
 &lt;/div&gt;
 &lt;/blockquote&gt;
 &lt;p&gt;Where &lt;span class="formula"&gt;&lt;i&gt;Oa&lt;/i&gt;&lt;/span&gt; is the contract address and &lt;span class="formula"&gt;&lt;i&gt;Ot&lt;/i&gt;&lt;/span&gt; is all of the topics.&lt;/p&gt;
 &lt;p&gt;In the context of &lt;em&gt;Solidity&lt;/em&gt;, &amp;quot;all&amp;quot; of the topics means the actual event signature, along with all the &lt;em&gt;indexed&lt;/em&gt; parameters to the event.&lt;/p&gt;
 &lt;p&gt;The &lt;span class="formula"&gt;&lt;i&gt;M&lt;/i&gt;&lt;/span&gt; is the filter bit generator. The filter 2048 bits long, and for each entry that is added, 3 bits need to be set.&lt;/p&gt;
 &lt;p&gt;Reading further, we learn that the bits to set are the &amp;quot;first three&amp;quot; pairs of bytes of the keccak256 hash &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;, modulo the length of the filter - 2048.&lt;/p&gt;
 &lt;p&gt;In the context of hashes, I usually assume &amp;quot;first&amp;quot; means from the left. That happens to be the case here, too. In other words, the &amp;quot;first&amp;quot; bytes are actually the bytes of most significance in big-endian order.&lt;/p&gt;
 &lt;p&gt;What's less clear, however, is which bits to set in the filter. The paper says:&lt;/p&gt;
 &lt;blockquote&gt;
 &lt;span class="formula"&gt;ℬ&lt;/span&gt; is the bit reference function such that &lt;span class="formula"&gt;ℬ&lt;sub&gt;&lt;i&gt;j&lt;/i&gt;&lt;/sub&gt;(&lt;i&gt;x&lt;/i&gt;)&lt;/span&gt; equals the bit of index &lt;span class="formula"&gt;&lt;i&gt;j&lt;/i&gt;&lt;/span&gt; (indexed from 0) in the byte array &lt;span class="formula"&gt;&lt;i&gt;x&lt;/i&gt;&lt;/span&gt;.&lt;/blockquote&gt;
 &lt;p&gt;Turns out after some trial and error that &amp;quot;index 0&amp;quot; means the &lt;em&gt;right&lt;/em&gt; end of the byte array. So when setting the bits, we have to count from the right side.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-friendly-recipe"&gt;
 &lt;h2&gt;The friendly recipe&lt;/h2&gt;
 &lt;p&gt;Now let's try to bring this down to a mortal level:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Take the keccak256 hash of the contract address&lt;/li&gt;
 &lt;li&gt;Take the keccak256 hash of the first topic value&lt;/li&gt;
 &lt;li&gt;Optionally, take the keccak256 hash of the remaining topic values (the indexed parameters of the event) &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/li&gt;
 &lt;li&gt;Instantiate a byte array with length 256&lt;/li&gt;
 &lt;li&gt;Now, &lt;em&gt;for each&lt;/em&gt; of the hashes:&lt;ol class="arabic"&gt;
 &lt;li&gt;Get the &lt;em&gt;big-endian&lt;/em&gt; integer of the &lt;em&gt;first two&lt;/em&gt; bytes of the hash&lt;/li&gt;
 &lt;li&gt;Calculate the 2048-modulo of that integer &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/li&gt;
 &lt;li&gt;&lt;em&gt;Bitwise or&lt;/em&gt; the filter bit at the index corresponding to the modulated value, counting from &lt;em&gt;right to left&lt;/em&gt; (i.e. the value &lt;span class="formula"&gt;0&lt;/span&gt; corresponds to the &lt;em&gt;rightmost bit&lt;/em&gt;).&lt;/li&gt;
 &lt;li&gt;Repeat 2 and 3 for the two next pairs of bytes of the hash.&lt;/li&gt;
 &lt;/ol&gt;
 &lt;/li&gt;
 &lt;/ol&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-code"&gt;
 &lt;h2&gt;The code&lt;/h2&gt;
 &lt;p&gt;All that taken into account, we can take a stab at implementing a filter generator:&lt;/p&gt;
 &lt;pre class="code python literal-block"&gt;
 &lt;span class="ln"&gt; 0 &lt;/span&gt;&lt;span class="c1"&gt;# external imports&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 1 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sha3&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 2 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 3 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 4 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LogBloom&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 5 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 6 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="fm"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 7 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;        &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;bytearray&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 8 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 9 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;10 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;element&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;11 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;element&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;12 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="ne"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'element must be bytes'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;13 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;        &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sha3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keccak_256&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;14 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;        &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;15 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;        &lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;16 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;17 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;18 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;19 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;byteorder&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'big'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;20 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;=&lt;/span&gt; &lt;span class="mh"&gt;0x07ff&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;21 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;255&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;22 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;23 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;            &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;|=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Let's say we'd want to check if an Solidity-defined event &lt;code&gt;Foo(uint256,bytes32)&lt;/code&gt; emitted by a contract at address &lt;code&gt;0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee&lt;/code&gt; exists in a filter.&lt;/p&gt;
 &lt;p&gt;Using the class above, we would have to do something like this:&lt;/p&gt;
 &lt;pre class="code python literal-block"&gt;
 &lt;span class="c1"&gt;# external imports&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sha3&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;some_block_getter&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# local imports&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="nn"&gt;the_above&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;LogBloom&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# all set bits in our filter must be set in theirs&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;filter_match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;theirs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ours&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ours&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ours&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;theirs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;ours&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;                        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="n"&gt;fltr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;LogBloom&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# add the contract address to the filter&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fromhex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sha3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keccak_256&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;address_element&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;fltr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;address_element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# create the topic signature for the event&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sha3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keccak_256&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s1"&gt;'Foo(uint256,bytes32)'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;topic&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# add the topic to the filter&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sha3&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keccak_256&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;topic&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;topic_element&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;fltr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;topic_element&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# get the filter from a block&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;some_block_getter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;block_bloom&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fromhex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'logsBloom'&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:])&lt;/span&gt; &lt;span class="c1"&gt;# assumes there is a 0x-prefix&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# check if it matches&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="n"&gt;match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;filter_match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block_bloom&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fltr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nb"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;match&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;/pre&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Remember the keccak hash used in Ethereum/Bitcoin is _not_ the official &lt;cite&gt;sha3&lt;/cite&gt; hash. The padding parameter is different. More on that &lt;a class="reference external" href="nft-tokenid-content.html#id8"&gt;here&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Solidity supports up to 3 indexed parameters, which maps to the &lt;em&gt;evm&lt;/em&gt; opcodes &lt;code&gt;LOGn&lt;/code&gt; where &lt;span class="formula"&gt;&lt;i&gt;n&lt;/i&gt; ∈ {1, 2, 3, 4}&lt;/span&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Since &lt;span class="formula"&gt;2&lt;sup&gt;11&lt;/sup&gt; = 2048&lt;/span&gt;, this is the same as zeroing out all bits above bit number 11; &lt;span class="formula"&gt;&lt;i&gt;x&lt;/i&gt;∧2048&lt;/span&gt;. In other words, the result is an &lt;em&gt;11 bit integer&lt;/em&gt;.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Code"></category><category term="crypto"></category><category term="ethereum"></category><category term="bloom filter"></category></entry><entry><title>Local npm repository</title><link href="docker-offline-3-npm.html" rel="alternate"></link><published>2021-05-25T11:47:00+02:00</published><updated>2021-05-25T11:47:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-05-25:docker-offline-3-npm.html</id><summary type="html">&lt;p class="first last"&gt;Feeding npm packages to your offline Docker setup&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;As expected, serving a local &lt;tt class="docutils literal"&gt;npm&lt;/tt&gt; repository is a &lt;em&gt;lot&lt;/em&gt; less straightforward than that for &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt; and &lt;tt class="docutils literal"&gt;python&lt;/tt&gt;.&lt;/p&gt;
 &lt;p&gt;The npm registry uses the &lt;code&gt;CouchDB&lt;/code&gt; nosql server as backend to resolve dependencies and serve resource metadata. The &lt;code&gt;npm&lt;/code&gt; CLI tool expects a corresponding json response to requests that cite the &lt;em&gt;name&lt;/em&gt; (and not the path) of the package.&lt;/p&gt;
 &lt;p&gt;Let's ask the &lt;a class="reference external" href="https://registry.npmjs.org"&gt;registry&lt;/a&gt; &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt; for the package &lt;tt class="docutils literal"&gt;ftp&lt;/tt&gt;, and have a look at the response (excerpt):&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;https://registry.npmjs.org/ftp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq
 &lt;span class="go"&gt;{&lt;/span&gt;
 &lt;span class="go"&gt;  &amp;quot;_id&amp;quot;: &amp;quot;ftp&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;  &amp;quot;_rev&amp;quot;: &amp;quot;113-89fe76508a7ece41b4c9a157114f966f&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;  &amp;quot;name&amp;quot;: &amp;quot;ftp&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;  &amp;quot;description&amp;quot;: &amp;quot;An FTP client module for node.js&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;  &amp;quot;dist-tags&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;    &amp;quot;latest&amp;quot;: &amp;quot;0.3.10&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;  },&lt;/span&gt;
 &lt;span class="go"&gt;  &amp;quot;versions&amp;quot;: {&lt;/span&gt;
 
 
 
 &lt;span class="go"&gt;    &amp;quot;0.3.10&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;name&amp;quot;: &amp;quot;ftp&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;version&amp;quot;: &amp;quot;0.3.10&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;author&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;name&amp;quot;: &amp;quot;Brian White&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;email&amp;quot;: &amp;quot;mscdex@mscdex.net&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;description&amp;quot;: &amp;quot;An FTP client module for node.js&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;main&amp;quot;: &amp;quot;./lib/connection&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;engines&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;node&amp;quot;: &amp;quot;&amp;gt;=0.8.0&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;dependencies&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;xregexp&amp;quot;: &amp;quot;2.0.0&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;readable-stream&amp;quot;: &amp;quot;1.1.x&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;scripts&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;test&amp;quot;: &amp;quot;node test/test.js&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;keywords&amp;quot;: [&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;ftp&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;client&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;transfer&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      ],&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;licenses&amp;quot;: [&lt;/span&gt;
 &lt;span class="go"&gt;        {&lt;/span&gt;
 &lt;span class="go"&gt;          &amp;quot;type&amp;quot;: &amp;quot;MIT&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;          &amp;quot;url&amp;quot;: &amp;quot;http://github.com/mscdex/node-ftp/raw/master/LICENSE&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;        }&lt;/span&gt;
 &lt;span class="go"&gt;      ],&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;repository&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;type&amp;quot;: &amp;quot;git&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;url&amp;quot;: &amp;quot;http://github.com/mscdex/node-ftp.git&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;bugs&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;url&amp;quot;: &amp;quot;https://github.com/mscdex/node-ftp/issues&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;homepage&amp;quot;: &amp;quot;https://github.com/mscdex/node-ftp&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;_id&amp;quot;: &amp;quot;ftp@0.3.10&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;_shasum&amp;quot;: &amp;quot;9197d861ad8142f3e63d5a83bfe4c59f7330885d&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;_from&amp;quot;: &amp;quot;https://github.com/mscdex/node-ftp/tarball/v0.3.10&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;_resolved&amp;quot;: &amp;quot;https://github.com/mscdex/node-ftp/tarball/v0.3.10&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;_npmVersion&amp;quot;: &amp;quot;1.4.28&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;_npmUser&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;name&amp;quot;: &amp;quot;mscdex&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;email&amp;quot;: &amp;quot;mscdex@mscdex.net&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;maintainers&amp;quot;: [&lt;/span&gt;
 &lt;span class="go"&gt;        {&lt;/span&gt;
 &lt;span class="go"&gt;          &amp;quot;name&amp;quot;: &amp;quot;mscdex&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;          &amp;quot;email&amp;quot;: &amp;quot;mscdex@mscdex.net&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;        }&lt;/span&gt;
 &lt;span class="go"&gt;      ],&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;dist&amp;quot;: {&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;shasum&amp;quot;: &amp;quot;9197d861ad8142f3e63d5a83bfe4c59f7330885d&amp;quot;,&lt;/span&gt;
 &lt;span class="go"&gt;        &amp;quot;tarball&amp;quot;: &amp;quot;https://registry.npmjs.org/ftp/-/ftp-0.3.10.tgz&amp;quot;&lt;/span&gt;
 &lt;span class="go"&gt;      },&lt;/span&gt;
 &lt;span class="go"&gt;      &amp;quot;directories&amp;quot;: {}&lt;/span&gt;
 &lt;span class="go"&gt;    }&lt;/span&gt;
 &lt;span class="go"&gt;  },&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;That looks a lot like embellished contents of the basic package.json set up by &lt;code&gt;npm init&lt;/code&gt;, except it's wrapped in a versions array. It also explicitly defines an &lt;em&gt;absolute&lt;/em&gt; url to a &lt;cite&gt;tarball&lt;/cite&gt; under the &lt;cite&gt;dist&lt;/cite&gt; key. &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/p&gt;
 &lt;div class="section" id="how-low-can-you-go"&gt;
 &lt;h2&gt;How low can you go&lt;/h2&gt;
 &lt;p&gt;It seems that all that's needed is a json index page served for the package name subpath of repository path, together with the tarball itself. We check this by putting together an utterly pointless package &lt;tt class="docutils literal"&gt;foobarbarbar&lt;/tt&gt;:&lt;/p&gt;
 &lt;p&gt;&lt;strong&gt;foobarbarbar/index.js&lt;/strong&gt;&lt;/p&gt;
 &lt;pre class="code javascript literal-block" id="foobarbarbar-index-js"&gt;
 &lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;exports&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="s1"&gt;'smth'&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;smth&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="kd"&gt;function&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;smth&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'foo'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;&lt;strong&gt;foobarbarbar/package.json&lt;/strong&gt;&lt;/p&gt;
 &lt;pre class="code json literal-block" id="foobarbarbar-package-json"&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foobarbarbar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;version&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1.0.0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo repo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;main&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;index.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;author&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo Bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;license&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;GPL3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Then make a tarball from those two files named &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;foobarbarbar-1.0.0.tgz&lt;/span&gt;&lt;/tt&gt;, take the sha1 sum of it,&lt;/p&gt;
 &lt;p&gt;Remembering our &lt;a class="reference external" href="docker-offline-2-python.html"&gt;virtual interface setup from the pip example&lt;/a&gt; we stick it behind our webserver (here with npm sub-path) and add our minimal version json wrapper:&lt;/p&gt;
 &lt;p&gt;&lt;strong&gt;package.json&lt;/strong&gt;&lt;/p&gt;
 &lt;pre class="code json literal-block" id="package-json"&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foobarbarbar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;versions&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;1.0.0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foobarbarbar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;version&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1.0.0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo repo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;main&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;index.js&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;author&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo Bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;license&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;GPL3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;dist&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                                 &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;shasum&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2ccd68498ef5f2bfa00f0e1e59f44686fdb296ee&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                                 &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;tarball&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;http://10.1.2.1/npm/foobarbarbar-1.0.0.tgz&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;/div&gt;
 &lt;div class="section" id="making-introductions"&gt;
 &lt;h2&gt;Making introductions&lt;/h2&gt;
 &lt;p&gt;The central trick here is to serve a json document as the &amp;quot;directory index&amp;quot; of the HTTP server. It's useful to remind ourselves at this point that we are &lt;em&gt;not&lt;/em&gt; setting up a &lt;em&gt;registry&lt;/em&gt; but a &lt;em&gt;repository&lt;/em&gt; that contains provisions for a &lt;em&gt;locked&lt;/em&gt; or &lt;em&gt;frozen&lt;/em&gt; dependency graph. The former would surely need some of the &lt;tt class="docutils literal"&gt;CouchDB&lt;/tt&gt; magic to resolve dependencies. Our assumption is that the latter can theoretically be realized using static files.&lt;/p&gt;
 &lt;p&gt;In other words; just as with the previous &lt;a class="reference external" href="docker-offline-2-python.html"&gt;python repository example&lt;/a&gt;, we don't try to handle dependency resolutions for pip, but merely serve the actual package files after dependences have been resolved.&lt;/p&gt;
 &lt;p&gt;With Apache Web Server, using the &lt;tt class="docutils literal"&gt;package.json&lt;/tt&gt; as the directory index is as easy as:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&amp;lt;Directory &amp;quot;/srv/http/npm&amp;quot;&amp;gt;
         DirectoryIndex package.json
 &amp;lt;/Directory&amp;gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Of course adjusting the &lt;cite&gt;Directory&lt;/cite&gt; path as needed to match the local setup.&lt;/p&gt;
 &lt;p&gt;Make sure that the tarball and the latter &lt;cite&gt;package.json&lt;/cite&gt; can be found in the &lt;cite&gt;foobarbarbar&lt;/cite&gt; virtual subfolder of the above &lt;cite&gt;Directory&lt;/cite&gt; directive path:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;ls&lt;span class="w"&gt; &lt;/span&gt;/srv/http/npm/foobarbarbar/
 &lt;span class="go"&gt;-rw-r--r-- 1 root root  351 May 24 18:38 foobarbarbar-1.0.0.tgz&lt;/span&gt;
 &lt;span class="go"&gt;-rw-r--r-- 1 root root  380 May 25 09:36 package.json&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Set the registry entry in your &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;~./npmrc&lt;/span&gt;&lt;/tt&gt; as follows:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="na"&gt;registry&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;http://10.1.2.1/npm&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Then restart the Apache server and give it a go:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;npm&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--verbose&lt;span class="w"&gt; &lt;/span&gt;foobarbarbar
 &lt;span class="go"&gt;npm verb cli [ &amp;#39;/usr/bin/node&amp;#39;, &amp;#39;/usr/bin/npm&amp;#39;, &amp;#39;install&amp;#39;, &amp;#39;--verbose&amp;#39;, &amp;#39;foobarbarbar&amp;#39; ]&lt;/span&gt;
 &lt;span class="go"&gt;npm info using npm@7.13.0&lt;/span&gt;
 &lt;span class="go"&gt;npm info using node@v16.1.0&lt;/span&gt;
 &lt;span class="go"&gt;[...]&lt;/span&gt;
 &lt;span class="go"&gt;npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/ 6ms&lt;/span&gt;
 &lt;span class="go"&gt;[...]&lt;/span&gt;
 &lt;span class="go"&gt;npm http fetch GET 200 http://10.1.2.1/npm/foobarbarbar/foobarbarbar-1.0.0.tgz 25ms&lt;/span&gt;
 &lt;span class="go"&gt;[...]&lt;/span&gt;
 &lt;span class="go"&gt;added 2 packages in 545ms&lt;/span&gt;
 &lt;span class="go"&gt;npm timing command:install Completed in 70ms&lt;/span&gt;
 &lt;span class="go"&gt;[...]&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="cache-dodging"&gt;
 &lt;h2&gt;Cache dodging&lt;/h2&gt;
 &lt;p&gt;Seriously, &lt;tt class="docutils literal"&gt;npm &lt;span class="pre"&gt;--help&lt;/span&gt;&lt;/tt&gt; leaves a lot to be desired. However, the online &lt;tt class="docutils literal"&gt;npm install&lt;/tt&gt; documentation does not yield any more clues as to whether it gives you an option of ignoring local cache like you can with &lt;tt class="docutils literal"&gt;pip &lt;span class="pre"&gt;--no-cache&lt;/span&gt;&lt;/tt&gt;. This is a major pain in the ass, as you have to remember to keep your &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;~/.npm&lt;/span&gt;&lt;/tt&gt; folder clean between each install attempt. Otherwise, changes you make to the repository won't be used by &lt;tt class="docutils literal"&gt;npm install&lt;/tt&gt;.&lt;/p&gt;
 &lt;p&gt;While testing, it pays to use a folder fairly close to the fs root, like a subfolder in &lt;cite&gt;tmp&lt;/cite&gt;. That way, you won't be thrown off by some stray &lt;cite&gt;node_modules&lt;/cite&gt; folder somewhere down the tree.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="think-locally-act-locally"&gt;
 &lt;h2&gt;Think locally, act locally&lt;/h2&gt;
 &lt;p&gt;Serving packages globally apparently comes down to this:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Pull the &lt;tt class="docutils literal"&gt;package.json&lt;/tt&gt; versions list from a proper registry.&lt;/li&gt;
 &lt;li&gt;Get the tarball&lt;/li&gt;
 &lt;li&gt;Transform the absolute package url to our host instead. (sigh)&lt;/li&gt;
 &lt;li&gt;(optional) Prune all the versions and metadata which is not needed (anything that's not in our minimal &lt;tt class="docutils literal"&gt;foobarbarbar&lt;/tt&gt; example above).&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;Obviously, this is an annoyingly large amount of work to code up parsing and retrieval for.&lt;/p&gt;
 &lt;p&gt;Incidentally, there is a very nice tool called &lt;a class="reference external" href="https://verdaccio.org"&gt;verdaccio&lt;/a&gt; which gets us most of the way there. &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt; Its &lt;em&gt;storage&lt;/em&gt; directory &lt;a class="footnote-reference" href="#footnote-4" id="footnote-reference-4"&gt;[4]&lt;/a&gt; uses exactly the directory structure we need. After retrieving a package collection using it as a proxy, getting the files in place is merely a case of copying &lt;a class="footnote-reference" href="#footnote-5" id="footnote-reference-5"&gt;[5]&lt;/a&gt; the files from the &lt;em&gt;storage&lt;/em&gt; directory to the corresponding webserver directory.&lt;/p&gt;
 &lt;p&gt;Looking at the &lt;tt class="docutils literal"&gt;package.json&lt;/tt&gt; versions wrapper saved by &lt;tt class="docutils literal"&gt;verdaccio&lt;/tt&gt;, however, we see that it still preserves the absolute path of the upstream registry. Sadly we are still left with doing the third task ourselves.&lt;/p&gt;
 &lt;p&gt;As parsing json is one of my least favorite things in the world, I won't include the code for this last step here. For now it's sufficient that we understand the minimums required for manually setting up and serving a static, local, offline npm repository. Regardless of the pain involved.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="making-it-personal"&gt;
 &lt;h2&gt;Making it personal&lt;/h2&gt;
 &lt;p&gt;Whether we massage jsons ourselves or lazily resort to &lt;tt class="docutils literal"&gt;verdaccio&lt;/tt&gt;, the final step we need to take is the same: Setting the registry url in the npm configuration of the docker image.&lt;/p&gt;
 &lt;p&gt;This is merely a case of setting the registry url in the &lt;a class="reference external" href="https://docs.npmjs.com/cli/v6/configuring-npm/npmrc#files"&gt;npmrc&lt;/a&gt; inside the container.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="gp"&gt;$ &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;npm&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;ls&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;etc/npmrc
 &lt;span class="go"&gt;globalconfig = &amp;quot;/usr/etc/npmrc&amp;quot;&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;To get at our manually provided &lt;tt class="docutils literal"&gt;foobarbarbar&lt;/tt&gt; package from before, the dockerfile will be (using archlinux):&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 &lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;nodejs&lt;span class="w"&gt; &lt;/span&gt;npm
 &lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-vp&lt;span class="w"&gt; &lt;/span&gt;/usr/etc
 &lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;registry=http:/10.1.2.1/npm&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/etc/npmrc
 &lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/root&lt;/span&gt;
 &lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;npm&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;foobarbarbar
 &lt;/pre&gt;&lt;/div&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;This was the registry address I had in my &lt;cite&gt;~/.npmrc&lt;/cite&gt; or whether I put it in myself.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;I have tried using a relative path instead, both with and without a leading &lt;tt class="docutils literal"&gt;/&lt;/tt&gt;, but in either case the install then errors our complaining that the lock file is &amp;quot;corrupt.&amp;quot; Whether the schema allows a base-url setting I do not know, as the schema documentation doesn't seem to be readily available.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;In fact, &lt;tt class="docutils literal"&gt;verdaccio&lt;/tt&gt; by itself solves the particular problem that we're trying to solve; providing an offline repository alternative. The task at hand, however, is to understand how to cope &lt;em&gt;without&lt;/em&gt; using other tools than what can be considered base infrastructure (i.e. a web server).&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-4" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Should be &lt;tt class="docutils literal"&gt;/etc/verdaccio/storage&lt;/tt&gt;; see &lt;tt class="docutils literal"&gt;/etc/verdaccio/config.yaml&lt;/tt&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-5" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-5"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;An alternative approach would be to set the storage path to point to the same folder as the web server is serving. However, since we need to mess with the tarball paths&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Offlining"></category><category term="docker"></category><category term="npm"></category><category term="nodejs"></category><category term="javascript"></category><category term="devops"></category></entry><entry><title>A village network protocol</title><link href="village-network-protocol.html" rel="alternate"></link><published>2021-05-19T10:40:00+02:00</published><updated>2021-05-19T10:40:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-05-19:village-network-protocol.html</id><summary type="html">&lt;p class="first last"&gt;What would truly peer-to-peer CRM backend look like?&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;What would truly peer-to-peer CRM backend look like? And at that, once that respected the right of the &amp;quot;customer&amp;quot; to choose how it wants to represent itself in the context?&lt;/p&gt;
 &lt;p&gt;Perhaps by Modeling the network on how villages treat strangers and their own, and how you would go about interacting with them.&lt;/p&gt;
 &lt;p&gt;Here's an initial stab at describing what such a meta-protocol could look like.&lt;/p&gt;
 &lt;!-- There is no truth. There is only perspective and predictability. --&gt;
 &lt;div class="section" id="terms"&gt;
 &lt;h2&gt;Terms&lt;/h2&gt;
 &lt;dl class="docutils"&gt;
 &lt;dt&gt;Individual&lt;/dt&gt;
 &lt;dd&gt;A subjective structure where the thing being described is the thing that described it.&lt;/dd&gt;
 &lt;dt&gt;User&lt;/dt&gt;
 &lt;dd&gt;A particular perception of an individual by an entity in the network.&lt;/dd&gt;
 &lt;dt&gt;Userspec&lt;/dt&gt;
 &lt;dd&gt;Data structure containing details about a user. May or may not be public.&lt;/dd&gt;
 &lt;dt&gt;Locals&lt;/dt&gt;
 &lt;dd&gt;A group of individuals acknowledging each other as part of a group, and who share views on users.&lt;/dd&gt;
 &lt;dt&gt;Local&lt;/dt&gt;
 &lt;dd&gt;A member of a group of locals, from the shared perspective of an individual and another local.&lt;/dd&gt;
 &lt;dt&gt;Localslist&lt;/dt&gt;
 &lt;dd&gt;The individuals making up a group of locals, from the perspective of a local.&lt;/dd&gt;
 &lt;dt&gt;Stranger&lt;/dt&gt;
 &lt;dd&gt;A user as percieved by a group of locals.&lt;/dd&gt;
 &lt;dt&gt;Strangerlist&lt;/dt&gt;
 &lt;dd&gt;All users that may speak to the locals, from the perspective of a local. To be shared among locals.&lt;/dd&gt;
 &lt;dt&gt;Strangerspec&lt;/dt&gt;
 &lt;dd&gt;Userspec created by a local describing a user. To be shared among locals.&lt;/dd&gt;
 &lt;dt&gt;Entity&lt;/dt&gt;
 &lt;dd&gt;An individual or any derived representation of it (user, stranger, local).&lt;/dd&gt;
 &lt;dt&gt;Peer&lt;/dt&gt;
 &lt;dd&gt;An entity coupled with connection information&lt;/dd&gt;
 &lt;dt&gt;Content address&lt;/dt&gt;
 &lt;dd&gt;SHA256 sum of arbitrary content&lt;/dd&gt;
 &lt;dt&gt;Key setup&lt;/dt&gt;
 &lt;dd&gt;May be an individual key, or a specification describing how to generate keys for a particular relation.&lt;/dd&gt;
 &lt;/dl&gt;
 &lt;/div&gt;
 &lt;div class="section" id="roles"&gt;
 &lt;h2&gt;Roles&lt;/h2&gt;
 &lt;div class="section" id="the-individuals"&gt;
 &lt;h3&gt;The individuals&lt;/h3&gt;
 &lt;p&gt;Are free to have their own opinion of what they are.&lt;/p&gt;
 &lt;p&gt;When interacting with others, they are free to choose what they say about what they are.&lt;/p&gt;
 &lt;p&gt;An individual initially &lt;em&gt;defines itself&lt;/em&gt; in the network.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-users"&gt;
 &lt;h3&gt;The users&lt;/h3&gt;
 &lt;p&gt;A user is a view of an individual as defined by one or more other individuals.&lt;/p&gt;
 &lt;p&gt;It may be how an individual presents itself to an particular individual or a group.&lt;/p&gt;
 &lt;p&gt;It may also be how an individual or a group presents another individual to itself.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-locals"&gt;
 &lt;h3&gt;The locals&lt;/h3&gt;
 &lt;p&gt;Locals are a group of individuals who wish to share a view of themselves and prejudice about others.&lt;/p&gt;
 &lt;p&gt;An individual will not be treated as a local unless it and its local counterpart considers it a local.&lt;/p&gt;
 &lt;p&gt;Locals may acknowledge strangers with or without proof.&lt;/p&gt;
 &lt;p&gt;If one local changes opinion about a stranger, all locals should change their opinon to the same.&lt;/p&gt;
 &lt;p&gt;If one local excludes a stranger, all locals should exclude the stranger.&lt;/p&gt;
 &lt;p&gt;You can't keep secrets in a village. If someone says something to another local, it must assume all locals will get to know what was said.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="the-strangers"&gt;
 &lt;h3&gt;The strangers&lt;/h3&gt;
 &lt;p&gt;A stranger is a user that's not considered local by a group of locals.&lt;/p&gt;
 &lt;p&gt;A stranger may not speak to a local until the local has decided it can.&lt;/p&gt;
 &lt;p&gt;If a stranger may speak to one local, it may speak all locals.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="identifiers"&gt;
 &lt;h2&gt;Identifiers&lt;/h2&gt;
 &lt;p&gt;The &lt;code&gt;individual_id&lt;/code&gt; is the identifier of an individual consists of the content address of some basic public data about the entity registering itself. The &lt;code&gt;individual_id&lt;/code&gt; is immutable.&lt;/p&gt;
 &lt;p&gt;Any local will refer to another local by its &lt;code&gt;individual_id&lt;/code&gt;.&lt;/p&gt;
 &lt;p&gt;A &lt;code&gt;user_id&lt;/code&gt; is the content address of an &lt;code&gt;individual_id&lt;/code&gt; together with the content address of some basic data describing the perception of the individual.&lt;/p&gt;
 &lt;p&gt;Locals describe stranger by a pair of &lt;code&gt;individual_id&lt;/code&gt; and &lt;code&gt;user_id&lt;/code&gt;. Let us call this &lt;code&gt;stranger_id&lt;/code&gt;. The &lt;code&gt;stranger_id&lt;/code&gt; is immutable.&lt;/p&gt;
 &lt;p&gt;The data represented by the &lt;code&gt;user_id&lt;/code&gt; in a &lt;code&gt;stranger_id&lt;/code&gt; is visible to locals, and may or may not be public.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="state-description"&gt;
 &lt;h2&gt;State description&lt;/h2&gt;
 &lt;p&gt;A user keeps a state towards other entities consisting of the following content-addresses, in the given order:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Previous state&lt;/li&gt;
 &lt;li&gt;Userspec&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;A local keeps a state towards its locals consisting of the following content-addresses, in the given order.&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;Previous state&lt;/li&gt;
 &lt;li&gt;Userspec&lt;/li&gt;
 &lt;li&gt;Localslist&lt;/li&gt;
 &lt;li&gt;Locals state&lt;/li&gt;
 &lt;li&gt;Strangerslist&lt;/li&gt;
 &lt;li&gt;Strangers state&lt;/li&gt;
 &lt;/ol&gt;
 &lt;p&gt;The state is represented by the content address of the concatenation of these content-addresses, in the given order.&lt;/p&gt;
 &lt;p&gt;Any state change will be signed by the key resolvable through the &lt;code&gt;individual_id&lt;/code&gt;.&lt;/p&gt;
 &lt;div class="section" id="previous-state"&gt;
 &lt;h3&gt;Previous state&lt;/h3&gt;
 &lt;p&gt;The content address of the previous state. Will be &lt;code&gt;0&lt;/code&gt; in the first state.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="userspec"&gt;
 &lt;h3&gt;Userspec&lt;/h3&gt;
 &lt;p&gt;The individual as it wants to be perceived in the current context. There is no guarantee that the individual will be viewed in this way by others.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="localslist"&gt;
 &lt;h3&gt;Localslist&lt;/h3&gt;
 &lt;p&gt;A lexiographically ordered array of &lt;code&gt;individual_id&lt;/code&gt; that define the individuals in a group of locals.&lt;/p&gt;
 &lt;p&gt;All locals must keep this list in the same state.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="locals-state"&gt;
 &lt;h3&gt;Locals state&lt;/h3&gt;
 &lt;p&gt;A local's last recorded state of each of the locals, in the same order as &lt;code&gt;Localslist&lt;/code&gt;.&lt;/p&gt;
 &lt;p&gt;The recorded state is the same content address as in the state description section above.&lt;/p&gt;
 &lt;p&gt;All locals must strive to keep this in the same state.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="strangerslist"&gt;
 &lt;h3&gt;Strangerslist&lt;/h3&gt;
 &lt;p&gt;A lexiographically ordered pointer array representing a local's opinion of which users that can communicate with locals. &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;Combination of &lt;code&gt;stranger_id&lt;/code&gt; and a &lt;em&gt;vote&lt;/em&gt; on the specific user.&lt;/p&gt;
 &lt;p&gt;The vote can be in one of three states:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;Value &lt;code&gt;0&lt;/code&gt;, meaning local has acknowledged the user, explicitly or by following other locals. We call this state &amp;quot;deferred.&amp;quot;&lt;/li&gt;
 &lt;li&gt;Value &lt;code&gt;-1&lt;/code&gt;, meaning a local has vetoed the user. We call this state &amp;quot;vetoed.&amp;quot;&lt;/li&gt;
 &lt;li&gt;Any other value, which is the content-address of the proof accepted by the local. &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;  We call this state &amp;quot;accepted.&amp;quot;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;Typically, the initial vote value is &lt;code&gt;0&lt;/code&gt;. The value may change at any time, but may never change back to &lt;code&gt;0&lt;/code&gt; once set to another value.&lt;/p&gt;
 &lt;p&gt;A user that is part of the list, but not vetoed, is referred to as &amp;quot;acknowledged.&amp;quot;&lt;/p&gt;
 &lt;p&gt;The Strangerslist state may be differ between locals, but they should strive to keep it the same.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="strangers-state"&gt;
 &lt;h3&gt;Strangers state&lt;/h3&gt;
 &lt;p&gt;The local's version of the current user state of the strangers, in the same order as the &lt;code&gt;Strangerlist&lt;/code&gt;.&lt;/p&gt;
 &lt;p&gt;The Strangers state may differ between locals, but they should strive to keep it the same.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="behavior-of-locals"&gt;
 &lt;h2&gt;Behavior of locals&lt;/h2&gt;
 &lt;p&gt;A local may announce itself as &lt;em&gt;active&lt;/em&gt; or &lt;em&gt;passive&lt;/em&gt;. This should be part of the &lt;code&gt;Userspec&lt;/code&gt; in the locals context.&lt;/p&gt;
 &lt;p&gt;Specifically, this affects the way it &amp;quot;votes&amp;quot; on strangers. A passive local:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;Will always &lt;em&gt;veto&lt;/em&gt; a user if &lt;em&gt;at least one active local&lt;/em&gt; has vetoed it.&lt;/li&gt;
 &lt;li&gt;Will always &lt;em&gt;accept&lt;/em&gt; a user if &lt;em&gt;all active locals&lt;/em&gt; have accepted it. &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;Typically, a passive local will be a bot, for example an always-online node that guarantees availability of the locals so users can request state changes.&lt;/p&gt;
 &lt;p&gt;There must be at least two active locals in any group of locals.&lt;/p&gt;
 &lt;div class="section" id="veto-dissent"&gt;
 &lt;h3&gt;Veto dissent&lt;/h3&gt;
 &lt;p&gt;If a local vetoes a stranger, then all locals should cut communication with that stranger.&lt;/p&gt;
 &lt;p&gt;Vetoes should be adopted by all locals.&lt;/p&gt;
 &lt;p&gt;If, however, other locals consider the veto to be issued in error, there are two possible outcomes:&lt;/p&gt;
 &lt;ol class="arabic simple"&gt;
 &lt;li&gt;The vetoing local(s) abandon(s) the veto. &lt;a class="footnote-reference" href="#footnote-4" id="footnote-reference-4"&gt;[4]&lt;/a&gt;&lt;/li&gt;
 &lt;li&gt;The vetoing individual(s) is/are excluded by the locals.&lt;/li&gt;
 &lt;/ol&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="network"&gt;
 &lt;h2&gt;Network&lt;/h2&gt;
 &lt;p&gt;Any entity needs to define and announce itself as a &lt;code&gt;peer&lt;/code&gt; in order to be available on the network.&lt;/p&gt;
 &lt;p&gt;A &amp;quot;peer&amp;quot; is the coupling of network connection information to an &lt;code&gt;individual_id&lt;/code&gt;.&lt;/p&gt;
 &lt;div class="section" id="discoverability"&gt;
 &lt;h3&gt;Discoverability&lt;/h3&gt;
 &lt;p&gt;Peers are kept in a &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Distributed_hash_table"&gt;DHT&lt;/a&gt;.&lt;/p&gt;
 &lt;p&gt;Peers must be discoverable by their locals affiliation. &lt;a class="footnote-reference" href="#footnote-5" id="footnote-reference-5"&gt;[5]&lt;/a&gt;&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="users"&gt;
 &lt;h3&gt;Users&lt;/h3&gt;
 &lt;p&gt;Users open single-request connections to other users to annonunce, look up and prune peers.&lt;/p&gt;
 &lt;p&gt;Users may request state changes for themselves, or relay such requests for other users.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="locals"&gt;
 &lt;h3&gt;Locals&lt;/h3&gt;
 &lt;p&gt;Locals keep open connections (streams) with each other.&lt;/p&gt;
 &lt;p&gt;Locals continually exchange state changes.&lt;/p&gt;
 &lt;p&gt;Locals should be able to audit the chain of state changes of other locals.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="between-locals-and-users"&gt;
 &lt;h3&gt;Between locals and users&lt;/h3&gt;
 &lt;p&gt;Strangers &lt;em&gt;acknowledged&lt;/em&gt; by locals open single-request connections to a local to announce state changes.&lt;/p&gt;
 &lt;p&gt;Locals may open single-request connections to a user to announce a change to the state of the &lt;code&gt;Userspec&lt;/code&gt; of a stranger. &lt;a class="footnote-reference" href="#footnote-6" id="footnote-reference-6"&gt;[6]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;Individuals are free to deny connections from non-locals, for example if the entity has been sending garbage requests. The implementation of such sanctions is deferred to the application layer.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="encryption"&gt;
 &lt;h2&gt;Encryption&lt;/h2&gt;
 &lt;p&gt;An individual may keep content totally private on the network.&lt;/p&gt;
 &lt;p&gt;A user may keep different encryption keys for different contexts.&lt;/p&gt;
 &lt;p&gt;If a stranger shares a key setup with a local, that setup will be shared among all locals. A local will expect to be able to communicate with a strangerr using this key setup.&lt;/p&gt;
 &lt;p&gt;The stranger must be able to change the key setup towards locals.&lt;/p&gt;
 &lt;p&gt;The locals may be able to change the key setup towards a stranger.&lt;/p&gt;
 &lt;p&gt;The protocol should allow anything from simple symmetric key exchanges to complex forward/backward secrecy schemes.&lt;/p&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Insert in this sorted list will face scaling issues when locals need to acknowledge a lot of users. The lists can be split into &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Bucket_%28computing%29"&gt;buckets&lt;/a&gt;, either statically (for example two-byte prefixes) or dynamically after crossing a certain threshold. It should not be necessary to store empty buckets.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;This proof may map to what's commonly known as &lt;em&gt;kyc information&lt;/em&gt;.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Perhaps with the further limitation that all active users must have accepted with the &lt;em&gt;same&lt;/em&gt; proof. Otherwise, how will the passive local choose which proof to accept.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-4" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Observe that since the vote state cannot change back to &lt;code&gt;0&lt;/code&gt;, &amp;quot;cancelling&amp;quot; the veto can only be done by adding proof.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-5" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-5"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;What exactly uniquely identifies a group of locals on the network in such a way that it cannot be hi-jacked or spoofed still needs to be thought through. Perhaps it can be a combination of a specific name with a signature from the &amp;quot;founders;&amp;quot; the first (two or more) locals in it.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-6" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-6"&gt;[6]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;A use-case here would be where the locals make a change for the user due to an out-of-band request from the user. This is &lt;em&gt;not&lt;/em&gt;  the &lt;code&gt;Strangerspec&lt;/code&gt;; it is the user's current &lt;code&gt;Userspec&lt;/code&gt; for the given context. Of course, it's up to the user to accept the change. It may in turn request a new state change to the locals to revert it.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Hygiene"></category><category term="p2p"></category><category term="protocol"></category><category term="network"></category><category term="crm"></category><category term="privacy"></category><category term="f2f"></category><category term="community"></category></entry><entry><title>The NFT token id as URI</title><link href="nft-tokenid-content.html" rel="alternate"></link><published>2021-05-08T19:14:00+02:00</published><updated>2021-05-10T09:18:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-05-08:nft-tokenid-content.html</id><summary type="html">&lt;p class="first last"&gt;How to embed asset references for NFTs that are independent of providers in the current standard.&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;Let's consider an NFT that works like a badge for participating in development of a software project.&lt;/p&gt;
 &lt;p&gt;This token is awarded as a proof that the task was completed.&lt;/p&gt;
 &lt;p&gt;To make things more fun, each NFT should have some unique, immutable content attached to it.&lt;/p&gt;
 &lt;p&gt;In other words, the properties of this token, once set, should never change.&lt;/p&gt;
 &lt;p&gt;Nor should they disappear.&lt;/p&gt;
 &lt;p&gt;So how do we refer to the artwork asset within the token standard?&lt;/p&gt;
 &lt;div class="section" id="it-was-acceptable-at-the-time"&gt;
 &lt;h2&gt;It was acceptable at the time&lt;/h2&gt;
 &lt;p&gt;The ERC721 standard is not explicit about where the assets that belong with the NFT can be discovered and resolved.&lt;/p&gt;
 &lt;p&gt;At the time when the standard was adopted by the Ethereum community, there were multiple &lt;em&gt;&amp;quot;[...] Alternatives considered: put all metadata for each asset on the blockchain (too expensive), use URL templates to query metadata parts (URL templates do not work with all URL schemes, especially P2P URLs), multiaddr network address (not mature enough).&amp;quot;&lt;/em&gt; Furthermore, they &lt;em&gt;&amp;quot;[...] considered an NFT representing ownership of a house, in this case metadata about the house (image, occupants, etc.) can naturally change.&amp;quot;&lt;/em&gt; &lt;a class="citation-reference" href="#eip721" id="citation-reference-1"&gt;[EIP721]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;A &amp;quot;changing house&amp;quot; doesn't sound quite like what we need. And anyway; if we stick a good old web2 URI in there, then that will end up on the great bonfire of dead links before long.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="image-schmimage"&gt;
 &lt;h2&gt;Image, schmimage&lt;/h2&gt;
 &lt;p&gt;To be honest, I find the presumption in the optional EIP721 metadata structure to be surprisingly short-sighted. It &lt;em&gt;specifically&lt;/em&gt; defines the asset as an image, and at the same time is presupposes that only a &lt;em&gt;single&lt;/em&gt; asset file will be used.&lt;/p&gt;
 &lt;p&gt;We may want to add &lt;em&gt;multiple&lt;/em&gt; sources, so this is another obstacle for us.&lt;/p&gt;
 &lt;p&gt;So how to get around this, while still playing nice with existing implementations out there? Two ideas come to mind:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;Embed a &lt;em&gt;thumbnail&lt;/em&gt; as a preview of the artwork using a &lt;code&gt;base64&lt;/code&gt; &lt;em&gt;data URI&lt;/em&gt; &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt; in the metadata. Stick &lt;code&gt;name&lt;/code&gt; and &lt;code&gt;description&lt;/code&gt; on it, and the schema is still fulfilled.&lt;/li&gt;
 &lt;li&gt;&lt;em&gt;Extend&lt;/em&gt; the structure with a list of &lt;em&gt;attachments&lt;/em&gt; that &lt;em&gt;our&lt;/em&gt; application layer knows about. Of course, each of these can have the same format as above.&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;In other words:&lt;/p&gt;
 &lt;pre class="code json literal-block"&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;data:image/gif;base64,R0lGODlhDgARAKEDAAAAAOjqAPP1APDw8CH+EUNyZWF0ZWQgd2l0aCBHSU1QACwAAAAADgARAAACO5wHqXdrClocodbUaGg34qoBHaJtl7VYIfqBZxepb5C6NBvbIlyucO6jPUIK1CMiSCYvDIVS4GhCJoYCADs=&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;attachments&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Bar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Bar image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data:image/gif;base64,&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Baz&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Baz image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;data:image/gif;base64,&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;/div&gt;
 &lt;div class="section" id="mirror-mirror"&gt;
 &lt;h2&gt;Mirror, mirror&lt;/h2&gt;
 &lt;p&gt;Since the asset reference shouldn't change, we can refer to it by its fingerprint or &lt;a class="reference external" href="https://en.wikipedia.org/wiki/Content-addressable_storage"&gt;content address&lt;/a&gt;. If we define that the resource can be looked up over HTTP by that fingerprint as its basename, then we are free to define and modify whatever list of mirrors for that resource that's valid for any point in time. The application layer would simply try the endpoints one after another.&lt;/p&gt;
 &lt;p&gt;We take the &lt;code&gt;sha2-256&lt;/code&gt; &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt; of the asset reference (the json file above, free of evil whitespace and newlines):&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;reference.json&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{ print $1; }&amp;#39;&lt;/span&gt;
 3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Imagine we had a mirror list of &lt;a class="reference external" href="https://foo.com"&gt;https://foo.com&lt;/a&gt; and &lt;a class="reference external" href="https://bar.com/baz/"&gt;https://bar.com/baz/&lt;/a&gt;. Then our application would try these urls in sequence, stopping at the first that returns a valid result:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;https://foo.com/3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551
 https://bar.com/baz/3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Once we receive the content, all we have to do is hash it ourselves and verify that the sum matches the basename of the URI. If it doesn't the result is of course not valid and we continue down the list, appropriately banning the mischievous server then throrougly harassing its admin.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="cast-away"&gt;
 &lt;h2&gt;Cast away&lt;/h2&gt;
 &lt;p&gt;Since our fingerprint is 32 bytes, it fits exactly inside the &lt;code&gt;tokenId&lt;/code&gt; (&lt;code&gt;uint256&lt;/code&gt;). Let's decide to big-endian numbers when converting (I find them easier to make sense of). In that case our hash from the reference turns into this modest number:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="c1"&gt;# python3&lt;/span&gt;
 &lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;hx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fromhex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="o"&gt;&amp;gt;&amp;gt;&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;from_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;byteorder&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;big&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
 &lt;span class="mi"&gt;28891040728719892888467057134569335350980764617882743994259054993630416573777&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;As long as we're composing the &lt;code&gt;evm&lt;/code&gt; inputs ourselves, we don't really have to worry about the integer representation in this particular case. But the interface is defined as an integer type, and other mortals may be using higher level interfaces, we have to be explicit about our choice.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="welcoming-mint"&gt;
 &lt;h2&gt;Welcoming mint&lt;/h2&gt;
 &lt;p&gt;Assume we have a method &lt;code&gt;mintTo(address _recipient, uint256 _tokenId)&lt;/code&gt; on our NFT contract. The Solidity signature of that contract is &lt;code&gt;edb20b7e&lt;/code&gt; &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-3"&gt;[3]&lt;/a&gt;. If I were to mint to myself then the input to the contract would be:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;edb20b7e000000000000000000000000185cbce7650ff7ad3b587e26b2877d95568805e33fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Broken down:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;signature:             edb20b7e
 address, zero-padded:  000000000000000000000000185cbce7650ff7ad3b587e26b2877d95568805e3
 token id:              3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;The corresponding web3.js code would look like:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;web3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;eth&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Contract&lt;/span&gt;&lt;span class="p"&gt;([...],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;0x...&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
 &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;methods&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;mintTo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;0x185cbce7650ff7ad3b587e26b2877d95568805e3&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;28891040728719892888467057134569335350980764617882743994259054993630416573777&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;call&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;To satisfy the &lt;cite&gt;tokenURI&lt;/cite&gt; method, we can generate a string that's prefix with sha256 as a &amp;quot;scheme&amp;quot; &lt;a class="footnote-reference" href="#footnote-4" id="footnote-reference-4"&gt;[4]&lt;/a&gt;. A bit of (unoptimized) solidity helps us out here:&lt;/p&gt;
 &lt;pre class="code solidity literal-block"&gt;
 &lt;span class="k"&gt;contract&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ni"&gt;NFT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="kt"&gt;uint256&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;token&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="kt"&gt;mapping&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uint256&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;uint256&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tokenIndex&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;// map token id to master token array index position
 &lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="p"&gt;[...]&lt;/span&gt;&lt;span class="w"&gt;
 
         &lt;/span&gt;&lt;span class="kt"&gt;function&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;tokenURI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uint256&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;_tokenId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;public&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pure&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;returns&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="kt"&gt;bytes32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;token_bytes&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="kt"&gt;bytes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;out&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="kt"&gt;uint8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="kt"&gt;uint256&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;c&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;token_bytes&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;bytes32&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;token&lt;span class="p"&gt;[&lt;/span&gt;tokenIndex&lt;span class="p"&gt;[&lt;/span&gt;_tokenId&lt;span class="p"&gt;]]);&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;out&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m-Decimal"&gt;64&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;7&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;s&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;h&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;a&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;5&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;6&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="m-Decimal"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;:&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 
                 &lt;/span&gt;c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;7&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="kt"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uint256&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;32&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;i&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uint8&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;_data&lt;span class="p"&gt;[&lt;/span&gt;i&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0xf0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="kt"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;c&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bytes1&lt;span class="p"&gt;(&lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x30&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;else&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;c&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bytes1&lt;span class="p"&gt;(&lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x57&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;uint8&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;_data&lt;span class="p"&gt;[&lt;/span&gt;i&lt;span class="p"&gt;])&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x0f&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="kt"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;c&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="m-Decimal"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bytes1&lt;span class="p"&gt;(&lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x30&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;else&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                                 &lt;/span&gt;out&lt;span class="p"&gt;[&lt;/span&gt;c&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="m-Decimal"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bytes1&lt;span class="p"&gt;(&lt;/span&gt;t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x57&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m-Decimal"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="kt"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;out&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;This will return &lt;code&gt;sha256:3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551&lt;/code&gt; for &lt;code&gt;tokenId&lt;/code&gt; &lt;code&gt;3fdfbfe3b510b69f90cd92618e4c1ec76cf8b9c330bc2da1922acda8f84f9551&lt;/code&gt; as input, provided that the &lt;code&gt;tokenId&lt;/code&gt; actually exists. That may seem a bit useless at first, but consider the scenario where we want to interface with other NFTs aswell. Or perhaps we are implementing a contract that optionally can support a static web2 URI in storage. By doing it this way, all bases are covered.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="decentralized-identifiers"&gt;
 &lt;h2&gt;Decentralized identifiers&lt;/h2&gt;
 &lt;p&gt;Even better would be to add redundancy with autonomous decentralized storage. However, networks like &lt;a class="reference external" href="https://ethswarm.org"&gt;Swarm&lt;/a&gt; and &lt;a class="reference external" href="https://ipfs.io"&gt;IPFS&lt;/a&gt; use their own hashing recipes. That means that for every network referenced, we'd have to define an &lt;em&gt;alternative&lt;/em&gt; in our reference structure.&lt;/p&gt;
 &lt;p&gt;Referencing the canonical &lt;code&gt;sha256&lt;/code&gt; aswell as the &lt;code&gt;Swarmhash&lt;/code&gt; for the same item could then look like this &lt;a class="footnote-reference" href="#footnote-5" id="footnote-reference-5"&gt;[5]&lt;/a&gt;:&lt;/p&gt;
 &lt;pre class="code json literal-block"&gt;
 &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;data:image/gif;base64,R0lGODlhDgARAKEDAAAAAOjqAPP1APDw8CH+EUNyZWF0ZWQgd2l0aCBHSU1QACwAAAAADgARAAACO5wHqXdrClocodbUaGg34qoBHaJtl7VYIfqBZxepb5C6NBvbIlyucO6jPUIK1CMiSCYvDIVS4GhCJoYCADs=&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;alternatives&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bzz:4b9149ee4550f2d786f9ba6584b79a30ee14ae05ff6e84a0f7c7561a14e3b779&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;description&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Foo image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
                         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sha256:d036a4ce7f929b632256225b2bebd81bdd558d3bfe3d96faae61db664708c16f&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
                 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;hr class="docutils" /&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Yes, they are valid URIs actually: &lt;a class="reference external" href="https://www.rfc-archive.org/getrfc.php?rfc=2397"&gt;https://www.rfc-archive.org/getrfc.php?rfc=2397&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Likely it would be prudent to start using the official &lt;code&gt;sha3&lt;/code&gt; instead of &lt;code&gt;sha2&lt;/code&gt; these days, also because the &lt;code&gt;sha2&lt;/code&gt; hash is not a builtin for &lt;code&gt;evm&lt;/code&gt;. But neither is &lt;code&gt;sha3&lt;/code&gt;. The &lt;code&gt;keccak256&lt;/code&gt; Bitcoin uses, which EVM has inherited, is a pre-cursor to the &lt;code&gt;keccak&lt;/code&gt; published as the &lt;em&gt;official&lt;/em&gt; &lt;code&gt;sha3&lt;/code&gt;. Still, &lt;code&gt;keccak256&lt;/code&gt; and &lt;code&gt;sha3&lt;/code&gt; is used interchangeably in opcode lists (and previously in &lt;a class="reference external" href="https://docs.soliditylang.org/en/v0.8.0/050-breaking-changes.html#functions"&gt;Solidity&lt;/a&gt; too). This has caused me quite a fair bit of confusion, I might add. Apart from it being ambiguous, the &lt;code&gt;keccak256&lt;/code&gt; tooling is also less common in the wild. Therefore &lt;code&gt;sha2&lt;/code&gt; seems like a safer bet for our experiments. It's not broken yet, after all.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;The hex result of &lt;code&gt;keccak256(&amp;quot;mintTo(address,uint256)&amp;quot;)&lt;/code&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-4" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-4"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;em&gt;Data URI&lt;/em&gt;  is of no use here, because the hash itself is just nondescript binary data. Luckily &lt;code&gt;&amp;lt;scheme&amp;gt;:&amp;lt;path&amp;gt;&lt;/code&gt; is still a valid URI.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-5" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-5"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Here the hashes represent the media content itself, not the reference. That's why the &lt;code&gt;sha256&lt;/code&gt; one is different than before.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils citation" frame="void" id="eip721" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#citation-reference-1"&gt;[EIP721]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://eips.ethereum.org/EIPS/eip-721"&gt;https://eips.ethereum.org/EIPS/eip-721&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Code"></category><category term="nft"></category><category term="evm"></category><category term="hash"></category><category term="key-value store"></category><category term="decentralized storage"></category></entry><entry><title>wasm and C - The bare necessities</title><link href="wasm-c-bare.html" rel="alternate"></link><published>2021-05-05T06:15:00+02:00</published><updated>2021-05-05T06:15:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-05-05:wasm-c-bare.html</id><summary type="html">&lt;p class="first last"&gt;The bare minimum needed for two-way communication between wasm and C&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;I am currently resuming my self-improvement task of learning Webassembly. Or, I should rather say, &lt;em&gt;using&lt;/em&gt; Webassembly.&lt;/p&gt;
 &lt;p&gt;Since I have some optimized small code chunks that I want to use in an embedded environment, it seemed sensible that the first step would be to establish two-way communication with C.&lt;/p&gt;
 &lt;p&gt;In my last outing around three years ago, I was using &lt;a class="reference external" href="https://emscripten.org/"&gt;Emscripten&lt;/a&gt; to bridge the gap. That tool adds quite a few bells and whistles, and doesn't quite yield that warm, fuzzy bare-metal rush. &lt;a class="reference external" href="https://emscripten.org/"&gt;Emscripten&lt;/a&gt; relies on &lt;a class="reference external" href="http://clang.org/"&gt;Clang&lt;/a&gt; and &lt;a class="reference external" href="https://llvm.org/"&gt;LLVM&lt;/a&gt;, all of which seem to have gotten their &lt;code&gt;wasm&lt;/code&gt; support built-in in the meantime (at least on my archlinux system). This it integrates nicely with &lt;a class="reference external" href="https://github.com/WebAssembly/wabt"&gt;wabt&lt;/a&gt; - the swiss-army knife of Webassembly.&lt;/p&gt;
 &lt;p&gt;So how far do we get with just &lt;a class="reference external" href="http://clang.org/"&gt;clang&lt;/a&gt;, &lt;a class="reference external" href="https://llvm.org/"&gt;LLVM&lt;/a&gt; and &lt;a class="reference external" href="https://github.com/WebAssembly/wabt"&gt;wabt&lt;/a&gt; ? Let's see if we at least can set up a code snippet which simply writes &lt;em&gt;&amp;quot;foobar&amp;quot;&lt;/em&gt; to memory. The host will write &lt;em&gt;&amp;quot;foo&amp;quot;&lt;/em&gt;, and &lt;code&gt;wasm&lt;/code&gt; will write &lt;em&gt;&amp;quot;bar&amp;quot;&lt;/em&gt;.&lt;/p&gt;
 &lt;div class="section" id="without-libc"&gt;
 &lt;h2&gt;Without libc&lt;/h2&gt;
 &lt;p&gt;This excellent &lt;a class="reference external" href="https://surma.dev/things/c-to-webassembly/"&gt;tutorial by Surma&lt;/a&gt; provides a good starting point. Go ahead and &lt;strong&gt;read that first&lt;/strong&gt;. This text is not a Webassembly primer, so the following will make a lot more sense if you do.&lt;/p&gt;
 &lt;p&gt;That setup still adds some magic. Namely, the &lt;em&gt;memory&lt;/em&gt; and &lt;em&gt;symbol table&lt;/em&gt; are here added by the &lt;em&gt;wasm linker&lt;/em&gt;. It would be even more fun to pass this from the host system instead.&lt;/p&gt;
 &lt;p&gt;And so we start:&amp;nbsp;&lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/p&gt;
 &lt;pre class="code c literal-block"&gt;
 &lt;span class="ln"&gt; 0 &lt;/span&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;quot;string.h&amp;quot;&lt;/span&gt;&lt;span class="cp"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 1 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 2 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;extern&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__heap_base&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 3 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;extern&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;call_me_sometime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 4 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 5 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;foo&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 6 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;__heap_base&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 7 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;'b'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 8 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 9 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;'r'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;10 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;call_me_sometime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;11 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Compiling this without linking gives us a hint on what needs to be defined.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;clang&lt;span class="w"&gt; &lt;/span&gt;--target&lt;span class="o"&gt;=&lt;/span&gt;wasm32&lt;span class="w"&gt; &lt;/span&gt;-nostdlib&lt;span class="w"&gt; &lt;/span&gt;-nostartfiles&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;bare.wasm&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;bare.c
 $&lt;span class="w"&gt; &lt;/span&gt;wasm-objdump&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;bare.wasm
 bare.wasm:&lt;span class="w"&gt;      &lt;/span&gt;file&lt;span class="w"&gt; &lt;/span&gt;format&lt;span class="w"&gt; &lt;/span&gt;wasm&lt;span class="w"&gt; &lt;/span&gt;0x1
 
 Section&lt;span class="w"&gt; &lt;/span&gt;Details:
 
 Type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;nil
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;i32&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;nil
 Import&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;memory&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pages:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;initial&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.__linear_memory
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;table&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;funcref&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;initial&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.__indirect_function_table
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;global&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;i32&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mutable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.__stack_pointer
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.call_me_sometime&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.call_me_sometime
 Function&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;
 Code&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;138&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;
 Custom:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;linking&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;symbol&lt;span class="w"&gt; &lt;/span&gt;table&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;F&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;func&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;binding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;global&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;hidden
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;G&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.__stack_pointer&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;global&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;undefined&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;binding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;global&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;default
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;D&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;__heap_base&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;undefined&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;binding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;global&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;default
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;F&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.call_me_sometime&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;func&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;undefined&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;binding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;global&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;default
 Custom:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;reloc.CODE&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;relocations&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;section:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;Code&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;R_wasm_GLOBAL_INDEX_LEB&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000007&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000099&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;symbol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.__stack_pointer&amp;gt;
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;R_wasm_GLOBAL_INDEX_LEB&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x00001c&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x0000ae&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;symbol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.__stack_pointer&amp;gt;
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;R_wasm_MEMORY_ADDR_SLEB&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000031&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x0000c3&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;symbol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;__heap_base&amp;gt;
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;R_wasm_FUNCTION_INDEX_LEB&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000073&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000105&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;symbol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.call_me_sometime&amp;gt;
 &lt;span class="w"&gt;   &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;R_wasm_GLOBAL_INDEX_LEB&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;offset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000086&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0x000118&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;symbol&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;env.__stack_pointer&amp;gt;
 Custom:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;producers&amp;quot;&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Using nodejs as the host, we check if we can instantiate a &lt;code&gt;WebAssembly&lt;/code&gt; object&lt;/p&gt;
 &lt;pre class="code javascript literal-block"&gt;
 &lt;span class="ln"&gt; 0 &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;fs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'fs'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 1 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 2 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;imports&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 3 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 4 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;function&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;init&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;&amp;nbsp;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 5 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'./bare.wasm'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 6 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WebAssembly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Module&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 7 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WebAssembly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Instance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;imports&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 8 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 9 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="nx"&gt;init&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Running this tells us we are apparently missing a property &lt;code&gt;env&lt;/code&gt; in the imports object.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;node&lt;span class="w"&gt; &lt;/span&gt;bare_naive.js
 /home/lash/src/tests/wasm/bare/bare_naive.js:8
 const&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;new&lt;span class="w"&gt; &lt;/span&gt;WebAssembly.Instance&lt;span class="o"&gt;(&lt;/span&gt;m,&lt;span class="w"&gt; &lt;/span&gt;imports&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
 &lt;span class="w"&gt;          &lt;/span&gt;^
 
 TypeError:&lt;span class="w"&gt; &lt;/span&gt;WebAssembly.Instance&lt;span class="o"&gt;()&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;Import&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;#0 module=&amp;quot;env&amp;quot; error: module is not an object or function&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;That seems to match with the &lt;code&gt;Import&lt;/code&gt; section in the &lt;code&gt;objdump&lt;/code&gt; output above. Let's stick the &lt;em&gt;memory&lt;/em&gt; and &lt;em&gt;table&lt;/em&gt; in there. &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;And let's make a bold guess that the callback function &lt;code&gt;call_me_sometime&lt;/code&gt; needs to go in there aswell.&lt;/p&gt;
 &lt;pre class="code javascript literal-block"&gt;
 &lt;span class="ln"&gt; 0 &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;fs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'fs'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 1 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 2 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WebAssembly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Memory&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="nx"&gt;initial&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="p"&gt;});&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 3 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WebAssembly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Table&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="nx"&gt;initial&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;element&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'anyfunc'&lt;/span&gt;&lt;span class="p"&gt;});&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 4 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;importsObj&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 5 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 6 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 7 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;__linear_memory&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 8 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;__indirect_function_table&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 9 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nx"&gt;call_me_sometime&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;10 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Uint8Array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;memory&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;buffer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;9&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;11 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="mh"&gt;0x66&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x6f&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;0x6f&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;12 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;debug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'heap is at: '&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;n&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;13 &lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'heap contains: '&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;TextDecoder&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nx"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;14 &lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;15 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;16 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;17 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;18 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;function&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;init&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;&amp;nbsp;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;19 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'./bare.wasm'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;20 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WebAssembly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Module&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;21 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WebAssembly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Instance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;importsObj&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;22 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;exports&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;foo&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;23 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;24 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="nx"&gt;init&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;The linker needs a little help from us for this:&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;Our callback function will not be available at link time, so we have to &lt;code&gt;--allow-undefined&lt;/code&gt; to promise that the host has got this covered.&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;--import-memory&lt;/code&gt; and &lt;code&gt;--import-table&lt;/code&gt; to enable us to get memory and symbol table from the host.&lt;/li&gt;
 &lt;li&gt;&lt;code&gt;--export=&amp;quot;foo&amp;quot;&lt;/code&gt; to make sure we only export exactly what we intend to from our &lt;code&gt;wasm&lt;/code&gt;.&lt;/li&gt;
 &lt;/ul&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;clang&lt;span class="w"&gt; &lt;/span&gt;--target&lt;span class="o"&gt;=&lt;/span&gt;wasm32&lt;span class="w"&gt; &lt;/span&gt;-nostdlib&lt;span class="w"&gt; &lt;/span&gt;-nostartfiles&lt;span class="w"&gt; &lt;/span&gt;-Wl,--no-entry&lt;span class="w"&gt; &lt;/span&gt;-Wl,--export&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-Wl,--import-memory&lt;span class="w"&gt; &lt;/span&gt;-Wl,--import-table&lt;span class="w"&gt; &lt;/span&gt;-Wl,--allow-undefined&lt;span class="w"&gt;  &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;bare.wasm&lt;span class="w"&gt; &lt;/span&gt;bare.c
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;And that should give us:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;node&lt;span class="w"&gt; &lt;/span&gt;bare.js
 heap&lt;span class="w"&gt; &lt;/span&gt;is&lt;span class="w"&gt; &lt;/span&gt;at:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;66560&lt;/span&gt;
 heap&lt;span class="w"&gt; &lt;/span&gt;contains:&lt;span class="w"&gt; &lt;/span&gt;foobar
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;This way of pointing to memory is of course grossly inadequate &lt;em&gt;and&lt;/em&gt; unsafe &lt;em&gt;and&lt;/em&gt; ridiculous for any purpose more advanced that this one. So some proper memory management would not be a bad thing.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="adding-libc"&gt;
 &lt;h2&gt;Adding libc&lt;/h2&gt;
 &lt;p&gt;And what do you know. In other news since last time I looked at this is the addition of &amp;quot;a libc for WebAssembly programs built on top of WASI system calls.&amp;quot; &lt;a class="citation-reference" href="#wasi-libc" id="citation-reference-1"&gt;[wasi-libc]&lt;/a&gt;. Let's see if we can add a slightly less manual way of handling memory with &lt;code&gt;malloc&lt;/code&gt; and &lt;code&gt;memcpy&lt;/code&gt;&lt;/p&gt;
 &lt;pre class="code c literal-block"&gt;
 &lt;span class="ln"&gt; 0 &lt;/span&gt;&lt;span class="cp"&gt;#ifdef HAVE_LIBC
 &lt;/span&gt;&lt;span class="ln"&gt; 1 &lt;/span&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;string.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 2 &lt;/span&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;stdlib.h&amp;gt;&lt;/span&gt;&lt;span class="cp"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 3 &lt;/span&gt;&lt;span class="cp"&gt;#endif
 &lt;/span&gt;&lt;span class="ln"&gt; 4 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 5 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;extern&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;__heap_base&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 6 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="k"&gt;extern&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;call_me_sometime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 7 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 8 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;foo&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt; 9 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;10 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="cp"&gt;#ifdef HAVE_LIBC
 &lt;/span&gt;&lt;span class="ln"&gt;11 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;12 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;malloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;13 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;memcpy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;bazbar&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;14 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="cp"&gt;#else
 &lt;/span&gt;&lt;span class="ln"&gt;15 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;unsigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;__heap_base&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;16 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;'b'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;17 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;18 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;'r'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;19 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="cp"&gt;#endif
 &lt;/span&gt;&lt;span class="ln"&gt;20 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;call_me_sometime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;21 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;22 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="cp"&gt;#ifdef HAVE_LIBC
 &lt;/span&gt;&lt;span class="ln"&gt;23 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;free&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;24 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="cp"&gt;#endif
 &lt;/span&gt;&lt;span class="ln"&gt;25 &lt;/span&gt;&lt;span class="cp"&gt;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="ln"&gt;26 &lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;As you see, we need a few more parameters for the compiler and linker at this point. The &lt;code&gt;--target=wasm32-unknown-wasi --sysroot /opt/wasi-libc .. /opt/wasi-libc/lib/wasm32-wasi/libc.a&lt;/code&gt; is needed to hook us up with headers and symbols for the libc.&lt;/p&gt;
 &lt;p&gt;My archlinux puts that sysroot in &lt;code&gt;/opt/wasi-libc&lt;/code&gt;, that may of course not be the case elsewhere.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;clang&lt;span class="w"&gt; &lt;/span&gt;-DHAVE_LIBC&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--target&lt;span class="o"&gt;=&lt;/span&gt;wasm32-unknown-wasi&lt;span class="w"&gt; &lt;/span&gt;--sysroot&lt;span class="w"&gt; &lt;/span&gt;/opt/wasi-libc&lt;span class="w"&gt; &lt;/span&gt;-nostdlib&lt;span class="w"&gt; &lt;/span&gt;-nostartfiles&lt;span class="w"&gt; &lt;/span&gt;-Wl,--no-entry&lt;span class="w"&gt; &lt;/span&gt;-Wl,--export&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-Wl,--import-memory&lt;span class="w"&gt; &lt;/span&gt;-Wl,--import-table&lt;span class="w"&gt; &lt;/span&gt;-Wl,--allow-undefined&lt;span class="w"&gt;  &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;bare.wasm&lt;span class="w"&gt; &lt;/span&gt;bare.c&lt;span class="w"&gt; &lt;/span&gt;/opt/wasi-libc/lib/wasm32-wasi/libc.a
 $&lt;span class="w"&gt; &lt;/span&gt;wasm-objdump&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;bare.wasm
 
 bare.wasm:&lt;span class="w"&gt;      &lt;/span&gt;file&lt;span class="w"&gt; &lt;/span&gt;format&lt;span class="w"&gt; &lt;/span&gt;wasm&lt;span class="w"&gt; &lt;/span&gt;0x1
 
 Section&lt;span class="w"&gt; &lt;/span&gt;Details:
 
 Type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;i32&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;nil
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;nil
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;type&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;i32&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;i32
 Import&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;memory&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pages:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;initial&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.memory
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;table&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;funcref&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;initial&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.__indirect_function_table
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;call_me_sometime&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;-&lt;span class="w"&gt; &lt;/span&gt;env.call_me_sometime
 Function&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;malloc&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;dlmalloc&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;free&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;dlfree&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;6&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;abort&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sig&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;sbrk&amp;gt;
 Global&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;global&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;i32&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mutable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;init&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i32&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;67072&lt;/span&gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;global&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;i32&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mutable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;__heap_base&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;init&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i32&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;67072&lt;/span&gt;
 Export&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;foo&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;global&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;__heap_base&amp;quot;&lt;/span&gt;
 Code&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;171&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;malloc&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;6984&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;dlmalloc&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;free&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1908&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;dlfree&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;6&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;abort&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;78&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;sbrk&amp;gt;
 Data&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;segment&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;memory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;init&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i32&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1024&lt;/span&gt;
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000400&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;6261&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;7a62&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;6172&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;00&lt;/span&gt;&lt;span class="w"&gt;                        &lt;/span&gt;bazbar.
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;segment&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;memory&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;500&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;init&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i32&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1032&lt;/span&gt;
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000408&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000418&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000428&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000438&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000448&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000458&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000468&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000478&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000488&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000498&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00004a8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00004b8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00004c8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00004d8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00004e8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00004f8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000508&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000518&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000528&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000538&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000548&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000558&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000568&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000578&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000588&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000598&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00005a8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00005b8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00005c8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00005d8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00005e8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;................
 &lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;00005f8:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0000&lt;/span&gt;&lt;span class="w"&gt;                                &lt;/span&gt;....
 Custom:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;call_me_sometime&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;foo&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;malloc&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;dlmalloc&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;free&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;dlfree&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;6&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;abort&amp;gt;
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;func&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;sbrk&amp;gt;
 Custom:
 &lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;producers&amp;quot;&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;What luxury. And of course, our &lt;code&gt;bare.wasm&lt;/code&gt; file just grew from 350 bytes to 10k...&lt;/p&gt;
 &lt;p&gt;We don't have to change our &lt;code&gt;javascript&lt;/code&gt; code at this point. Simply run again, and get:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;node&lt;span class="w"&gt; &lt;/span&gt;bare.js
 heap&lt;span class="w"&gt; &lt;/span&gt;is&lt;span class="w"&gt; &lt;/span&gt;at:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;67088&lt;/span&gt;
 heap&lt;span class="w"&gt; &lt;/span&gt;contains:&lt;span class="w"&gt; &lt;/span&gt;foobazbar
 &lt;/pre&gt;&lt;/div&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;code&gt;__heap_base&lt;/code&gt; will be set by default by the wasm environment, and is thus available as an external symbol.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;After linking the memory symbol meeds to be called &lt;code&gt;memory&lt;/code&gt; instead of &lt;code&gt;__linear_memory&lt;/code&gt; for some reason. Thus we add both here for clarity.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils citation" frame="void" id="wasi-libc" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#citation-reference-1"&gt;[wasi-libc]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;&lt;a class="reference external" href="https://github.com/WebAssembly/wasi-libc"&gt;https://github.com/WebAssembly/wasi-libc&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Code"></category><category term="wasm"></category><category term="c"></category><category term="clang"></category><category term="llvm"></category></entry><entry><title>Proving what you link to</title><link href="web-snapshot.html" rel="alternate"></link><published>2021-05-03T14:22:00+02:00</published><updated>2024-06-19T16:21:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-05-03:web-snapshot.html</id><summary type="html">&lt;p class="first last"&gt;Generating proof of a web resource when you read and share&lt;/p&gt;
 </summary><content type="html">&lt;p&gt;When we send a link to someone, we are trusting that whoever is behind that link will serve that someone the content we actually saw. Give or take an ad or two.&lt;/p&gt;
 &lt;p&gt;Also, when we bookmark the link for later retrieval. We are trusting that the entity will be serving that content at any time in the future that you may want it.&lt;/p&gt;
 &lt;p&gt;This may not be a huge problem if the page is merely a list of &lt;a class="reference external" href="https://thoughtcatalog.com/clint-conway/2016/08/50-of-the-funniest-dead-baby-jokes-of-all-time/"&gt;dead baby jokes&lt;/a&gt; &lt;a class="footnote-reference" href="#footnote-6" id="footnote-reference-1"&gt;[6]&lt;/a&gt;. They are objectively funny, of course. But you can also get along without them.&lt;/p&gt;
 &lt;p&gt;But what of the case of formal, scientific texts we may depend on, and that use citations from the web as part of their source material? Usually, they refer to a source by &lt;em&gt;link&lt;/em&gt; and &lt;em&gt;date of retrieval&lt;/em&gt;. This is not of much use unless the actual source and/or render they saw at that time also is available.&lt;/p&gt;
 &lt;p&gt;That may not always be the case.&lt;/p&gt;
 &lt;div class="section" id="take-care-of-your-shelf"&gt;
 &lt;h2&gt;Take care of your shelf&lt;/h2&gt;
 &lt;blockquote&gt;
 &amp;quot;No worries, the &lt;a class="reference external" href="https://archive.org"&gt;Wayback Machine&lt;/a&gt; has me covered.&amp;quot;&lt;/blockquote&gt;
 &lt;p&gt;Yes. But no. The Wayback Machine is a (thus far) centralized entity that depend on a few idealists and donations to keep going. If they stop to keep going, they depend on passing the buck to someone else. If that someone is evil, they may take it and rewrite history to suit themselves. If that someone else cannot be found, it becomes a garbage collection blip on Bezos' &lt;a class="reference external" href="https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336"&gt;infrastructure monopoly&lt;/a&gt; dashboard. &lt;a class="footnote-reference" href="#footnote-1" id="footnote-reference-2"&gt;[1]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;That aside, sources like Wayback Machine are like libraries. Libraries are, of course, essential. Not only because they serve as one of the pillars of democracy, providing free access to knowledge for everyone. They are also essential because it's simply not very practical for you to pre-emptively own all the books that you may at some point want to read. Let alone ask around in your neighborhood if they happen to have a copy (although a crowd-sourced library app sounds like a fun decentralization project to explore).&lt;/p&gt;
 &lt;p&gt;You may however want to keep a copy of the books you &lt;em&gt;depend&lt;/em&gt; on, and the ones you &lt;em&gt;really like&lt;/em&gt;. Just to make really sure you have it available when you want it. Then, if some New Public Management clowns get the chance to gut public infrastructure where you live, or someone starts a good old-fashioned fascist book burning, you have yourself covered.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="a-lack-of-friction"&gt;
 &lt;h2&gt;A lack of friction&lt;/h2&gt;
 &lt;p&gt;Yes, stuff may disappear on the web. Just as books may.&lt;/p&gt;
 &lt;p&gt;On the web that stuff can get &lt;em&gt;rewritten&lt;/em&gt;. Books may be rewritten, too. &lt;a class="footnote-reference" href="#footnote-2" id="footnote-reference-3"&gt;[2]&lt;/a&gt; The previous editions of the books will still exist as independent physical objects until they degrade, as long as something is not actively done to them. But so too with data on storage media. If it is not renewed or copied during that lifetime it may degrade until it becomes illegible. And of course, it may also simply be deleted. And without the smell of smoke at that.&lt;/p&gt;
 &lt;p&gt;That's the difference that seems to leap out when using this imagery. How  &lt;em&gt;easy&lt;/em&gt; it is to change and destroy stuff at scale on the web compared with the real world of books. And how inconspicuously it can happen, without &lt;em&gt;anyone&lt;/em&gt; noticing. And for those who notice, it is very hard to prove what has changed, unless you have a copy. &lt;a class="footnote-reference" href="#footnote-5" id="footnote-reference-4"&gt;[5]&lt;/a&gt;&lt;/p&gt;
 &lt;p&gt;So what can we do? Copies and proofs are definitely keywords here. Fortunately, copying is what computers are all about. And making a cryptographical proof of what you see is easy enough these days, too. The tricky bit is to build &lt;em&gt;credibility&lt;/em&gt; around that proof. But let's stick our head in the sand and start with the easy part and see where it takes us.&lt;/p&gt;
 &lt;/div&gt;
 &lt;div class="section" id="look-no-head"&gt;
 &lt;h2&gt;Look, no head&lt;/h2&gt;
 &lt;!-- .. WARNING::
 
         Nerdy zone ahead --&gt;
 &lt;p&gt;A good start is to dump the document source to disk, then calculating and storing the sum of it.&lt;/p&gt;
 &lt;p&gt;In many cases this will be insufficient, though, as many sites populates the DOM through scripts, either in part or in full. As the use-case here is humans voting on what they see is what they get, human aids will be needed. In other words, we need to render the page to store what we actually see.&lt;/p&gt;
 &lt;p&gt;Printing to PDF from the browser is an option, but that is really difficult to automate. Fortunately, modern browser engines provide command line access to rendering. Since I mostly use &lt;a class="reference external" href="https://brave.com/"&gt;Brave Browser&lt;/a&gt; these days, we'll use &lt;em&gt;headless Chromium&lt;/em&gt; here.&lt;/p&gt;
 &lt;p&gt;In addition to source, sum and rendering, we should also include a copy of the request headers for good measure.&lt;/p&gt;
 &lt;p&gt;Thus, we end up with something like this.&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 &lt;span class="ch"&gt;#!/bin/bash
 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;WEBSHOT_OUTPUT_DIR&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="p"&gt;/tmp&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;using&lt;span class="w"&gt; &lt;/span&gt;outdir&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;+e&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# prepare
 &lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nv"&gt;TZ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;UTC&lt;span class="w"&gt; &lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%Y%m%d%H%M&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;t&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nb"&gt;pushd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$t&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# store raw outputs
 &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;url.txt&lt;span class="w"&gt;
 &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-I&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;headers.txt&lt;span class="w"&gt;
 &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;GET&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;contents.txt&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;z&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;contents.txt&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$z&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;contents.txt.sha256&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;h&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$z&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;'{ print $1; }'&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$title&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;title&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$h&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;using&lt;span class="w"&gt; &lt;/span&gt;title&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$title&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# rendered snapshot
 &lt;/span&gt;chromium&lt;span class="w"&gt; &lt;/span&gt;--headless&lt;span class="w"&gt; &lt;/span&gt;--print-to-pdf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;n&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;_&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;h&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;mv&lt;span class="w"&gt; &lt;/span&gt;output.pdf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$n&lt;/span&gt;.pdf&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# store result
 &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;$title&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;tar&lt;span class="w"&gt; &lt;/span&gt;-zcvf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;$title&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;$n&lt;/span&gt;&lt;span class="s2"&gt;.tar.gz&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="c1"&gt;# clean up
 &lt;/span&gt;&lt;span class="nb"&gt;popd&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e
 &lt;/pre&gt;
 &lt;/div&gt;
 &lt;div class="section" id="what-does-this-mean"&gt;
 &lt;h2&gt;What does this mean&lt;/h2&gt;
 &lt;p&gt;Let's sum up what information we've managed to store with this operation.&lt;/p&gt;
 &lt;ul class="simple"&gt;
 &lt;li&gt;We have a copy of the unrendered source. It may or may not include all the information we want to store.&lt;/li&gt;
 &lt;li&gt;We have a fingerprint of that source.&lt;/li&gt;
 &lt;li&gt;We have a copy of the headers we were served when retrieving the document. &lt;a class="footnote-reference" href="#footnote-3" id="footnote-reference-5"&gt;[3]&lt;/a&gt;&lt;/li&gt;
 &lt;li&gt;We have a image copy of what we actually saw when visiting the page.&lt;/li&gt;
 &lt;li&gt;We have a date and time for retrieval (file attributes).&lt;/li&gt;
 &lt;/ul&gt;
 &lt;p&gt;To link the headers together with the visual copy, we could sum the header file and image file aswell, put those sums together with the content sum in a deterministic order, and calculate the sum of those sums. E.g. &lt;a class="footnote-reference" href="#footnote-4" id="footnote-reference-6"&gt;[4]&lt;/a&gt;&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;contents.txt.sha256&lt;span class="w"&gt; &lt;/span&gt;sums.txt
 $&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;headers.txt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{ print $1; }&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;sums.txt
 $&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pdf&lt;span class="w"&gt; &lt;/span&gt;file&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{ print $1; }&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;sums.txt
 $&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;sums.txt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{ print $1; }&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;topsum.txt
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;If we now sign &lt;em&gt;this&lt;/em&gt; sum, we are confirming that for this particuar resource:&lt;/p&gt;
 &lt;blockquote&gt;
 &amp;quot;This was the source. These were the headers for that source. This is how that source served in that manner looked for &lt;strong&gt;ME&lt;/strong&gt; at the time&amp;quot;&lt;/blockquote&gt;
 &lt;/div&gt;
 &lt;div class="section" id="proving-links-in-this-post"&gt;
 &lt;h2&gt;Proving links in this post&lt;/h2&gt;
 &lt;p&gt;This post makes use of several external links to articles. So as a final step, let's eat our own dogfood and add proofs for them.&lt;/p&gt;
 &lt;table border="1" class="docutils"&gt;
 &lt;caption&gt;Article retrieval proofs&lt;/caption&gt;
 &lt;colgroup&gt;
 &lt;col width="34%" /&gt;
 &lt;col width="22%" /&gt;
 &lt;col width="22%" /&gt;
 &lt;col width="22%" /&gt;
 &lt;/colgroup&gt;
 &lt;thead valign="bottom"&gt;
 &lt;tr&gt;&lt;th class="head"&gt;Link&lt;/th&gt;
 &lt;th class="head"&gt;Content&lt;/th&gt;
 &lt;th class="head"&gt;Header&lt;/th&gt;
 &lt;th class="head"&gt;Image&lt;/th&gt;
 &lt;/tr&gt;
 &lt;/thead&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td&gt;&lt;a class="reference external" href="https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336"&gt;https://gizmodo.com/i-tried-to-block-amazon-from-my-life-it-was-impossible-1830565336&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/0c657ec25e55e72702b0473f62e6d632dece992485f67c407ed1f748b3f40bc2.txt"&gt;0c657ec2&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/360125fa513b8db8380eb2b62c4479164baa5b48d8544b2242bcc7305bad0de4.txt"&gt;360125fa&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/a2584ca07ba16b15b9fad315a767cbb0c4b7124abdfd578cab2afb4dce9d1971.pdf"&gt;a2584ca0&lt;/a&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.thelocal.se/20111109/37244/"&gt;https://www.thelocal.se/20111109/37244/&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/be8741ff91c6e3eac760a3a4652b57f47bce92fa9b617662296c2ab5b3c5fe31.txt"&gt;be8741ff&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/6c20e7678a0235467687425b9818718ab143fd477afee2087d7f79c933abdc75.txt"&gt;6c20e767&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/f1c557b9555149dde570976ed956ffb17d17b99cea5f2651020f66408dacf301.pdf"&gt;f1c557b9&lt;/a&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/"&gt;https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/595a07b85e6b75a41fe0880c8538f15c4b6da5770db230d986efa2e080ca479a.txt"&gt;595a07b8&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/f2aa95b42c5420cb11ccbf1cb084d5e5ccc3fc6cd575c51e9ee473b9df19c890.txt"&gt;f2aa95b4&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/6a610fa69d4909cc9aa1dcb7105203cc50c1bcf26733411964f95cbcbdf37eb5.pdf"&gt;6a610fa6&lt;/a&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;a class="reference external" href="https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html"&gt;https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/5a806be410da82986a85f68658279234c1a5cf3bb6dc55da137d5274dc722f26.txt"&gt;5a806be4&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/078d9fe6be070de0d378a6e903f8558a7da4917cba5c95ca453ae1936541e4f6.txt"&gt;078d9fe6&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/2d0f0e69e7c8b6ffe9ff8ffc8702a78d6a2d46ab1edd4123e84eb211171d6cde.pdf"&gt;2d0f0e69&lt;/a&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;&lt;td&gt;&lt;a class="reference external" href="https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/"&gt;https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/a8c6b61cec2cce1a58bcf7a65091a6c2e8510ca5fa17f2d242d286f087d95cd5.txt"&gt;a8c6b61c&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/3ba9094b295a898cfe8cba256f4ebf65ef98ff05bb3034e048e517d52fc13d33.txt"&gt;3ba9094b&lt;/a&gt;&lt;/td&gt;
 &lt;td&gt;&lt;a class="reference external" href="misc/web_snapshot/eb76a87f224b0e4f4e5d1673b1505aaeee28d8ad03ce544656b6f5ac7c4d9983.pdf"&gt;eb76a87f&lt;/a&gt;&lt;/td&gt;
 &lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;p&gt;Clicking on the &amp;quot;image&amp;quot; links, we see that thanks to the recent ubiquity of cookie nag boxes screaming &amp;quot;accept all&amp;quot; at you, those very boxes are now blocking the content we want to get at. So we will need more work to get to where we want by automation. But it's s start.&lt;/p&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-1" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-2"&gt;[1]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Early 2021 survey puts Amazon at one-third of the global market share. &lt;a class="reference external" href="https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/"&gt;https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/&lt;/a&gt;&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-2" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-3"&gt;[2]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;2011 sparked a &lt;a class="reference external" href="https://www.thelocal.se/20111109/37244/"&gt;controversy&lt;/a&gt; around Astrid Lindgren's Pippi Longstockings. Echoing those viewpoints, the books were edited in 2015 during which some alleged &amp;quot;racist&amp;quot; content was altered. The rabid rightwing media later spun a &lt;a class="reference external" href="https://www.breitbart.com/europe/2017/07/19/swedens-libraries-pulp-traditional-children-books-racist-phrases/"&gt;false tale of mass purges&lt;/a&gt; of Pippi books around one single swedish library's decision to throw out copies of the originial &amp;quot;racist&amp;quot; versions. Case-in-point, a public statement in which the library tries to justify its actions is no longer available on their website, and has to be retrieved by the Wayback Machine. &lt;a class="reference external" href="https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html"&gt;https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html&lt;/a&gt; (in swedish)&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-3" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-5"&gt;[3]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Well actually, not quite. We did the same request twice, but they were two separate requests. Using a single request would improve the script.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-4" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-6"&gt;[4]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;We use the hex representation for clarity here. A proper tool would convert the hex values to bytes before calculating sum on them.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-5" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-4"&gt;[5]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;Another important difference is that the book does not need to be &lt;em&gt;interpreted&lt;/em&gt; by a &lt;em&gt;machine&lt;/em&gt; in order to make sense for a human. Availability of tooling is definitely an equally important topic in this discussion. However this post limits focus to the actual data itself.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;table class="docutils footnote" frame="void" id="footnote-6" rules="none"&gt;
 &lt;colgroup&gt;&lt;col class="label" /&gt;&lt;col /&gt;&lt;/colgroup&gt;
 &lt;tbody valign="top"&gt;
 &lt;tr&gt;&lt;td class="label"&gt;&lt;a class="fn-backref" href="#footnote-reference-1"&gt;[6]&lt;/a&gt;&lt;/td&gt;&lt;td&gt;How ironic; that site is gone. And I don't have a snapshot. What a shame. I've changed the link to some different baby jokes. The original url was &lt;a class="reference external" href="https://dead-baby-joke.com"&gt;https://dead-baby-joke.com&lt;/a&gt;.&lt;/td&gt;&lt;/tr&gt;
 &lt;/tbody&gt;
 &lt;/table&gt;
 &lt;/blockquote&gt;
 &lt;!-- .. _data availability:  https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.433.3480 --&gt;
 &lt;!-- https://web.archive.org/web/20170710185409/https://www.botkyrka.se/arkiv/nyhetsarkiv/nyheter-startsida/2017-07-10-angaende-uttalanden-av-journalisten-janne-josefsson-om-bibliotek-botkyrka.html &lt;- not available on link, no result on search https://www.mhpbooks.com/the-trouble-with-pippi/ --&gt;
 &lt;/div&gt;
 </content><category term="Archiving"></category><category term="web"></category><category term="hash"></category><category term="chromium"></category></entry><entry><title>Local python repository</title><link href="docker-offline-2-python.html" rel="alternate"></link><published>2021-04-26T07:55:00+02:00</published><updated>2021-04-26T07:55:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-04-26:docker-offline-2-python.html</id><summary type="html">&lt;p class="first last"&gt;Feeding python packages to your offline Docker setup&lt;/p&gt;
 </summary><content type="html">&lt;!-- .. CAUTION::
 
    This is a purely technical article. You should probably be `this geeky &lt;https://g33k.holbrook.no/c18beedd&gt;`_ to continue reading. --&gt;
 &lt;p&gt;In the previous part of this series we were able to connect a Docker network to a virtual interface on our host, neither of which have access to the internet. That means we are ready to host content for the container locally. And we will start out with creating a local Python repository.&lt;/p&gt;
 &lt;div class="section" id="packaging-the-packages"&gt;
 &lt;h2&gt;Packaging the packages&lt;/h2&gt;
 &lt;p&gt;I'll be so bold as to assume that you are using &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt; to manage your packages. It gives you not only the option to &lt;em&gt;install&lt;/em&gt; packages, but merely &lt;em&gt;download&lt;/em&gt; them to storage aswell. So let's do that and try to serve the packages.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;download&lt;span class="w"&gt; &lt;/span&gt;faker
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 Successfully&lt;span class="w"&gt; &lt;/span&gt;downloaded&lt;span class="w"&gt; &lt;/span&gt;faker&lt;span class="w"&gt; &lt;/span&gt;python-dateutil&lt;span class="w"&gt; &lt;/span&gt;six&lt;span class="w"&gt; &lt;/span&gt;text-unidecode
 $&lt;span class="w"&gt; &lt;/span&gt;ls
 Faker-8.1.0-py3-none-any.whl&lt;span class="w"&gt;  &lt;/span&gt;python_dateutil-2.8.1-py2.py3-none-any.whl&lt;span class="w"&gt;  &lt;/span&gt;six-1.15.0-py2.py3-none-any.whl&lt;span class="w"&gt;  &lt;/span&gt;text_unidecode-1.3-py2.py3-none-any.whl
 $&lt;span class="w"&gt; &lt;/span&gt;python&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;RangeHTTPServer
 &lt;/pre&gt;&lt;/div&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;python&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;venv&lt;span class="w"&gt; &lt;/span&gt;.venv
 $&lt;span class="w"&gt; &lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;.venv/bin/activate
 &lt;span class="o"&gt;(&lt;/span&gt;.venv&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--index&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000&lt;span class="w"&gt; &lt;/span&gt;faker
 Looking&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;indexes:&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000
 ERROR:&lt;span class="w"&gt; &lt;/span&gt;Could&lt;span class="w"&gt; &lt;/span&gt;not&lt;span class="w"&gt; &lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;a&lt;span class="w"&gt; &lt;/span&gt;version&lt;span class="w"&gt; &lt;/span&gt;that&lt;span class="w"&gt; &lt;/span&gt;satisfies&lt;span class="w"&gt; &lt;/span&gt;the&lt;span class="w"&gt; &lt;/span&gt;requirement&lt;span class="w"&gt; &lt;/span&gt;faker&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;from&lt;span class="w"&gt; &lt;/span&gt;versions:&lt;span class="w"&gt; &lt;/span&gt;none&lt;span class="o"&gt;)&lt;/span&gt;
 ERROR:&lt;span class="w"&gt; &lt;/span&gt;No&lt;span class="w"&gt; &lt;/span&gt;matching&lt;span class="w"&gt; &lt;/span&gt;distribution&lt;span class="w"&gt; &lt;/span&gt;found&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;faker
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Dag nabbit. Apparently not that simple. And indeed, if we read up on the spec for &lt;a class="reference external" href="https://www.python.org/dev/peps/pep-0503"&gt;PEP 503 spec for Simple Repository API&lt;/a&gt;, we learn that we are going to stick those package files under directories named after the packages.&lt;/p&gt;
 &lt;p&gt;Well, nothing that a bit of bash scripting can't sort out:&lt;/p&gt;
 &lt;pre class="code bash literal-block"&gt;
 &lt;span class="ch"&gt;#!/usr/bin/env
 &lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;s&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;usage: pep503.sh &amp;lt;source_dir&amp;gt; &amp;lt;dest_dir&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;df&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;*.whl&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;basename&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$df&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;s/^\(.*\)-[[:digit:]]*\.[[:digit:]].*&lt;/span&gt;$&lt;span class="s2"&gt;/\1/g&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[:upper:]&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[:lower:]&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;_&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/&lt;span class="nv"&gt;$pd&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$df&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/&lt;span class="nv"&gt;$pd&lt;/span&gt;/&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;df&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;*.tar.gz&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;basename&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$df&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;s/^\(.*\)-[[:digit:]]*\.[[:digit:]].*&lt;/span&gt;$&lt;span class="s2"&gt;/\1/g&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[:upper:]&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[:lower:]&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;_&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/&lt;span class="nv"&gt;$pd&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$df&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/&lt;span class="nv"&gt;$pd&lt;/span&gt;/&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;df&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;*.zip&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;f&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;basename&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$df&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;&lt;span class="nv"&gt;pd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;s/^\(.*\)-[[:digit:]]*\.[[:digit:]].*&lt;/span&gt;$&lt;span class="s2"&gt;/\1/g&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[:upper:]&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[:lower:]&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;_&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/&lt;span class="nv"&gt;$pd&lt;/span&gt;&lt;span class="w"&gt;
         &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$df&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;/&lt;span class="nv"&gt;$pd&lt;/span&gt;/&lt;span class="w"&gt;
 &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
 &lt;/pre&gt;
 &lt;p&gt;Armed with this, let's try again:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;sh&lt;span class="w"&gt; &lt;/span&gt;/home/lash/bin/shell/pep503.sh&lt;span class="w"&gt; &lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;packages
 $&lt;span class="w"&gt; &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;packages
 $&lt;span class="w"&gt; &lt;/span&gt;bash&lt;span class="w"&gt; &lt;/span&gt;/home/lash/bin/shell/pep503.sh&lt;span class="w"&gt; &lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;packages
 mkdir:&lt;span class="w"&gt; &lt;/span&gt;created&lt;span class="w"&gt; &lt;/span&gt;directory&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/text-unidecode&amp;#39;&lt;/span&gt;
 &lt;span class="s1"&gt;&amp;#39;./text_unidecode-1.3-py2.py3-none-any.whl&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/text-unidecode/text_unidecode-1.3-py2.py3-none-any.whl&amp;#39;&lt;/span&gt;
 mkdir:&lt;span class="w"&gt; &lt;/span&gt;created&lt;span class="w"&gt; &lt;/span&gt;directory&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/six&amp;#39;&lt;/span&gt;
 &lt;span class="s1"&gt;&amp;#39;./six-1.15.0-py2.py3-none-any.whl&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/six/six-1.15.0-py2.py3-none-any.whl&amp;#39;&lt;/span&gt;
 mkdir:&lt;span class="w"&gt; &lt;/span&gt;created&lt;span class="w"&gt; &lt;/span&gt;directory&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/python-dateutil&amp;#39;&lt;/span&gt;
 &lt;span class="s1"&gt;&amp;#39;./python_dateutil-2.8.1-py2.py3-none-any.whl&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/python-dateutil/python_dateutil-2.8.1-py2.py3-none-any.whl&amp;#39;&lt;/span&gt;
 mkdir:&lt;span class="w"&gt; &lt;/span&gt;created&lt;span class="w"&gt; &lt;/span&gt;directory&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/faker&amp;#39;&lt;/span&gt;
 &lt;span class="s1"&gt;&amp;#39;./Faker-8.1.0-py3-none-any.whl&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packages/faker/Faker-8.1.0-py3-none-any.whl&amp;#39;&lt;/span&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;packages/&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f
 packages/faker/Faker-8.1.0-py3-none-any.whl
 packages/python-dateutil/python_dateutil-2.8.1-py2.py3-none-any.whl
 packages/six/six-1.15.0-py2.py3-none-any.whl
 packages/text-unidecode/text_unidecode-1.3-py2.py3-none-any.whl
 k&lt;span class="w"&gt; &lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;python&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;RangeHTTPServer
 &lt;/pre&gt;&lt;/div&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--index&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/packages&lt;span class="w"&gt; &lt;/span&gt;faker
 Looking&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;indexes:&lt;span class="w"&gt; &lt;/span&gt;http://localhost:8000/packages
 Collecting&lt;span class="w"&gt; &lt;/span&gt;faker
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 Successfully&lt;span class="w"&gt; &lt;/span&gt;installed&lt;span class="w"&gt; &lt;/span&gt;faker-8.1.0&lt;span class="w"&gt; &lt;/span&gt;python-dateutil-2.8.1&lt;span class="w"&gt; &lt;/span&gt;six-1.15.0&lt;span class="w"&gt; &lt;/span&gt;text-unidecode-1.3
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="extra-prepping"&gt;
 &lt;h2&gt;Extra prepping&lt;/h2&gt;
 &lt;p&gt;There are some basic packages you will most always need, and which &lt;tt class="docutils literal"&gt;pip&lt;/tt&gt; often will expect to find in at least one of its available repositories, even regardless of whether its installed or not. If you don't have this on your local offline repository when the internet goes &lt;em&gt;poof&lt;/em&gt;, then that will block any build you are trying to do.&lt;/p&gt;
 &lt;p&gt;Let's make sure we have the packages around all the time:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;download&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;setuptools&lt;span class="w"&gt; &lt;/span&gt;setuptools-markdown&lt;span class="w"&gt; &lt;/span&gt;wheel
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 $&lt;span class="w"&gt; &lt;/span&gt;bash&lt;span class="w"&gt; &lt;/span&gt;/home/lash/bin/shell/pep503.sh&lt;span class="w"&gt; &lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;packages
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="choosing-a-server"&gt;
 &lt;h2&gt;Choosing a server&lt;/h2&gt;
 &lt;p&gt;As I tend to favor the classics, I still use &lt;tt class="docutils literal"&gt;Apache Web Server&lt;/tt&gt; to host stuff to my local environment. One practical (if not all too safe) thing about it is that it will automatically bind to all interfaces. So to make the repository available, we simply link or add the &lt;tt class="docutils literal"&gt;packages&lt;/tt&gt; directory to the virtual root, and restart the server e.g. with&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;httpd
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Of course you can use any HTTP server you like, as long as you know how to bind it to the virtual interface.&lt;/p&gt;
 &lt;p&gt;To verify that the service is providing what's needed, simply point your web browser to the location, e.g.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;lynx&lt;span class="w"&gt; &lt;/span&gt;http://10.1.2.1/packages/
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="serving-the-packages"&gt;
 &lt;h2&gt;Serving the packages&lt;/h2&gt;
 &lt;p&gt;Now that we are all prepped, the next step is to do install packages from within a Docker container.&lt;/p&gt;
 &lt;p&gt;Let's start with an Archlinux base image with basic python provisions.&lt;/p&gt;
 &lt;pre class="code docker literal-block"&gt;
 &lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;archlinux:latest&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-Sy&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;--noconfirm&lt;span class="w"&gt; &lt;/span&gt;python&lt;span class="w"&gt; &lt;/span&gt;python-pip
 &lt;/pre&gt;
 &lt;p&gt;Build and tag it with &lt;tt class="docutils literal"&gt;pythonbase&lt;/tt&gt;, and then let's add the Dockerfile to test the repository:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 
 &lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--index&lt;span class="w"&gt; &lt;/span&gt;http://10.1.2.1/packages&lt;span class="w"&gt; &lt;/span&gt;--trusted-host&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.2.1&lt;span class="w"&gt; &lt;/span&gt;faker
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Now, turn internet off (and lights, too, if you'd like some extra suspense), and build the second file.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;.
 Sending&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;context&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;Docker&lt;span class="w"&gt; &lt;/span&gt;daemon&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;.072kB
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 Successfully&lt;span class="w"&gt; &lt;/span&gt;installed&lt;span class="w"&gt; &lt;/span&gt;faker-8.1.0&lt;span class="w"&gt; &lt;/span&gt;python-dateutil-2.8.1&lt;span class="w"&gt; &lt;/span&gt;six-1.15.0&lt;span class="w"&gt; &lt;/span&gt;text-unidecode-1.3
 Removing&lt;span class="w"&gt; &lt;/span&gt;intermediate&lt;span class="w"&gt; &lt;/span&gt;container&lt;span class="w"&gt; &lt;/span&gt;2c10ffcdf3ad
 ---&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;1ba83bb8e111
 Successfully&lt;span class="w"&gt; &lt;/span&gt;built&lt;span class="w"&gt; &lt;/span&gt;1ba83bb8e111
 &lt;/pre&gt;&lt;/div&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;/blockquote&gt;
 &lt;/div&gt;
 </content><category term="Offlining"></category><category term="docker"></category><category term="python"></category><category term="devops"></category></entry><entry><title>The routing to freedom</title><link href="docker-offline-1-routing.html" rel="alternate"></link><published>2021-04-26T07:54:00+02:00</published><updated>2021-04-26T07:54:00+02:00</updated><author><name>Louis Holbrook</name></author><id>tag:None,2021-04-26:docker-offline-1-routing.html</id><summary type="html">&lt;p class="first last"&gt;How to not be forced being online when forced to use docker&lt;/p&gt;
 </summary><content type="html">&lt;!-- .. CAUTION::
 
    This is a purely technical article. You should probably be `this geeky &lt;https://g33k.holbrook.no/8319a926&gt;`_ to continue reading. --&gt;
 &lt;p&gt;Five years ago I decided that I wanted to be able to work from anywhere, anytime. Four years ago, that promise was kept. In part.&lt;/p&gt;
 &lt;p&gt;I do not need to go to an office somewhere. I can work outside in a park if I want to. I can ride on to a new town every day. I only ever need to bring my trusty old &lt;a class="reference external" href="https://www.tuxedocomputers.com/en"&gt;Tuxedo Laptop&lt;/a&gt; whereever I go.&lt;/p&gt;
 &lt;p&gt;All of this as true, as long as there is internet available. And, as it turns out, &lt;em&gt;good&lt;/em&gt; internet available.&lt;/p&gt;
 &lt;p&gt;This has become especially obvious to me once I started to work with a project that involves a collection of microservices contained in a Docker environment, which also makes extensive use of custom packages that change frequently alongside the development process. Turns out, every time I want to rebuild my cluster of containers when sitting in the sun in a park, I need my LTE modem to play along. If it doesn't, then a single package that can't be reached will thwart the build.&lt;/p&gt;
 &lt;p&gt;This does not feel much like freedom after all. So let's see how we can serve all of these locally from our host instead.&lt;/p&gt;
 &lt;p&gt;First of all, we have to be able to reach our local host from the Docker containers. This is less straightforward than it may seem at first. The most obvious solution is to use the &lt;tt class="docutils literal"&gt;host&lt;/tt&gt; network driver, but this exposes your &lt;em&gt;whole&lt;/em&gt; localhost interface and routes to internet, too. Aside from the security issues that raises, it can also also trick you into assuming that some resources are available when they in fact will not be when you move on to a different environment. What we want is to &lt;em&gt;block&lt;/em&gt; access to internet, while &lt;em&gt;choosing&lt;/em&gt; which services to let the Docker container use.&lt;/p&gt;
 &lt;p&gt;Once we have this in place, we want to create local repositories for all the stuff we otherwise need to download. In this particular case, that means a &lt;strong&gt;Docker&lt;/strong&gt; repository, a &lt;strong&gt;Python&lt;/strong&gt; repository, a &lt;strong&gt;nodejs&lt;/strong&gt; repository and a &lt;strong&gt;linux&lt;/strong&gt; repository. We'll use &lt;a class="reference external" href="https://archlinux.org"&gt;Archlinux&lt;/a&gt; for this exercise, because that's been my home environment for the last four years.&lt;/p&gt;
 &lt;p&gt;In fact, having your own mirror of all these and anything else you base most of your work on is not only a good idea for the purpose of &lt;em&gt;offlining&lt;/em&gt; in itself, but wasting on bandwidth for items you've already downloaded hundreds of times is not exactly a nod to climate awareness either. And even more importantly, ensuring &lt;em&gt;availability&lt;/em&gt; of software is something we should all participate in, and not merely defer to a couple of git repository giants.&lt;/p&gt;
 &lt;blockquote&gt;
 &lt;/blockquote&gt;
 &lt;!--  --&gt;
 &lt;blockquote&gt;
 &lt;/blockquote&gt;
 &lt;div class="section" id="reaching-the-local-host"&gt;
 &lt;h2&gt;Reaching the local host&lt;/h2&gt;
 &lt;p&gt;&lt;em&gt;Local host&lt;/em&gt; not &lt;em&gt;localhost&lt;/em&gt;, mind you. Which means we need a different interface to connect to. And since we are not wiring up in any physical sense, a virtual interface seems to be the reasonable way to go.&lt;/p&gt;
 &lt;p&gt;First, let's prepare a base Docker layer with some tools that you should never leave home without.&lt;/p&gt;
 &lt;div class="section" id="prepare-the-docker-image"&gt;
 &lt;h3&gt;Prepare the docker image&lt;/h3&gt;
 &lt;pre class="code docker literal-block"&gt;
 &lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;archlinux:latest&lt;/span&gt;&lt;span class="w"&gt;
 
 &lt;/span&gt;&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-Sy&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\
 &lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;--noconfirm&lt;span class="w"&gt; &lt;/span&gt;gnu-netcat&lt;span class="w"&gt; &lt;/span&gt;socat&lt;span class="w"&gt; &lt;/span&gt;inetutils&lt;span class="w"&gt; &lt;/span&gt;iproute2
 &lt;/pre&gt;
 &lt;p&gt;Let's build this as an image called &lt;tt class="docutils literal"&gt;archbase&lt;/tt&gt;. Provided the content above is a file called &lt;tt class="docutils literal"&gt;Dockerfile.archbase&lt;/tt&gt; in your current directory:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;archbase&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;Dockerfile.archbase&lt;span class="w"&gt; &lt;/span&gt;.
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="set-up-network-interfaces"&gt;
 &lt;h3&gt;Set up network interfaces&lt;/h3&gt;
 &lt;p&gt;Bring up a virtual interface&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;ip&lt;span class="w"&gt; &lt;/span&gt;link&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;foo&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dummy
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Find the subnet of the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;no-internet&lt;/span&gt;&lt;/tt&gt; Docker network. This network driver is a builtin that provides exactly what it advertises.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;network&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;no-internet
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 &lt;span class="s2"&gt;&amp;quot;Config&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Subnet&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.1.1.0/24&amp;quot;&lt;/span&gt;,
 &lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Gateway&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.1.1.1&amp;quot;&lt;/span&gt;
 &lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
 &lt;span class="o"&gt;]&lt;/span&gt;
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Assign an IP address to the dummy interface in a &lt;em&gt;different&lt;/em&gt; subnet than the one the Docker network uses.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;ip&lt;span class="w"&gt; &lt;/span&gt;addr&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.2.1/24&lt;span class="w"&gt; &lt;/span&gt;dev&lt;span class="w"&gt; &lt;/span&gt;foo
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="traverse-the-firewall"&gt;
 &lt;h3&gt;Traverse the firewall&lt;/h3&gt;
 &lt;p&gt;Find the bridge used by the Docker container. Look for an ip address that matches the gateway of the docker network config.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;ip&lt;span class="w"&gt; &lt;/span&gt;addr&lt;span class="w"&gt; &lt;/span&gt;ls
 &lt;span class="m"&gt;17&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;br-d4ddb68f9938:&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;BROADCAST,MULTICAST,UP,LOWER_UP&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;mtu&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1500&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;qdisc&lt;span class="w"&gt; &lt;/span&gt;noqueue&lt;span class="w"&gt; &lt;/span&gt;state&lt;span class="w"&gt; &lt;/span&gt;UP&lt;span class="w"&gt; &lt;/span&gt;group&lt;span class="w"&gt; &lt;/span&gt;default
 link/ether&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;02&lt;/span&gt;:42:9c:1a:58:d2&lt;span class="w"&gt; &lt;/span&gt;brd&lt;span class="w"&gt; &lt;/span&gt;ff:ff:ff:ff:ff:ff
 inet&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.1.1/24&lt;span class="w"&gt; &lt;/span&gt;brd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.1.255&lt;span class="w"&gt; &lt;/span&gt;scope&lt;span class="w"&gt; &lt;/span&gt;global&lt;span class="w"&gt; &lt;/span&gt;br-d4ddb68f9938
 valid_lft&lt;span class="w"&gt; &lt;/span&gt;forever&lt;span class="w"&gt; &lt;/span&gt;preferred_lft&lt;span class="w"&gt; &lt;/span&gt;forever
 inet6&lt;span class="w"&gt; &lt;/span&gt;fe80::42:9cff:fe1a:58d2/64&lt;span class="w"&gt; &lt;/span&gt;scope&lt;span class="w"&gt; &lt;/span&gt;link
 valid_lft&lt;span class="w"&gt; &lt;/span&gt;forever&lt;span class="w"&gt; &lt;/span&gt;preferred_lft&lt;span class="w"&gt; &lt;/span&gt;forever
 &lt;span class="o"&gt;[&lt;/span&gt;...&lt;span class="o"&gt;]&lt;/span&gt;
 &lt;span class="m"&gt;7614&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;foo:&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;BROADCAST,NOARP,UP,LOWER_UP&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;mtu&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1500&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;qdisc&lt;span class="w"&gt; &lt;/span&gt;noqueue&lt;span class="w"&gt; &lt;/span&gt;master&lt;span class="w"&gt; &lt;/span&gt;br-496e528b928c&lt;span class="w"&gt; &lt;/span&gt;state&lt;span class="w"&gt; &lt;/span&gt;UNKNOWN&lt;span class="w"&gt; &lt;/span&gt;group&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;qlen&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1000&lt;/span&gt;
 link/ether&lt;span class="w"&gt; &lt;/span&gt;1a:50:53:2b:96:98&lt;span class="w"&gt; &lt;/span&gt;brd&lt;span class="w"&gt; &lt;/span&gt;ff:ff:ff:ff:ff:ff
 inet&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.3.1/24&lt;span class="w"&gt; &lt;/span&gt;scope&lt;span class="w"&gt; &lt;/span&gt;global&lt;span class="w"&gt; &lt;/span&gt;foo
 valid_lft&lt;span class="w"&gt; &lt;/span&gt;forever&lt;span class="w"&gt; &lt;/span&gt;preferred_lft&lt;span class="w"&gt; &lt;/span&gt;forever
 inet6&lt;span class="w"&gt; &lt;/span&gt;fe80::1850:53ff:fe2b:9698/64&lt;span class="w"&gt; &lt;/span&gt;scope&lt;span class="w"&gt; &lt;/span&gt;link
 valid_lft&lt;span class="w"&gt; &lt;/span&gt;forever&lt;span class="w"&gt; &lt;/span&gt;preferred_lft&lt;span class="w"&gt; &lt;/span&gt;forever
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Add the virtual interface to the bridge.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;ip&lt;span class="w"&gt; &lt;/span&gt;link&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;foo&lt;span class="w"&gt; &lt;/span&gt;master&lt;span class="w"&gt; &lt;/span&gt;br-d4ddb68f9938
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Long story short, the previous step will make the traffic from the container reach the &lt;tt class="docutils literal"&gt;INPUT&lt;/tt&gt; chain in &lt;tt class="docutils literal"&gt;iptables&lt;/tt&gt;. Now we can make an exception for incoming traffic from the &lt;tt class="docutils literal"&gt;&lt;span class="pre"&gt;no-internet&lt;/span&gt;&lt;/tt&gt; Docker bridge.&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;iptables&lt;span class="w"&gt; &lt;/span&gt;-I&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--source&lt;span class="w"&gt; &lt;/span&gt;br-d4ddb68f9938&lt;span class="w"&gt; &lt;/span&gt;--destination&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.2.1/24&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
 &lt;/pre&gt;&lt;/div&gt;
 &lt;/div&gt;
 &lt;div class="section" id="verify"&gt;
 &lt;h3&gt;Verify&lt;/h3&gt;
 &lt;p&gt;Provided you don't have any other hurdles in your local &lt;tt class="docutils literal"&gt;ìptables&lt;/tt&gt; setup, a port on device &lt;tt class="docutils literal"&gt;foo&lt;/tt&gt; should be reachable from the docker container. We can use socat to check.&lt;/p&gt;
 &lt;p&gt;On local host:&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;socat&lt;span class="w"&gt; &lt;/span&gt;TCP4-LISTEN:8000,bind&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.1.2.1,reuseaddr&lt;span class="w"&gt; &lt;/span&gt;-
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Start the docker container with shell prompt&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;docker&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--network&lt;span class="w"&gt; &lt;/span&gt;no-internet&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;archbase&lt;span class="w"&gt; &lt;/span&gt;/bin/bash
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;The moment of truth&lt;/p&gt;
 &lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;$&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;bar&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;socat&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;TCP4:10.1.2.1:8000
 &lt;/pre&gt;&lt;/div&gt;
 &lt;p&gt;Spoiler: &lt;tt class="docutils literal"&gt;bar&lt;/tt&gt; should pop up on the local host side.&lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt;
 </content><category term="Offlining"></category><category term="docker"></category><category term="networking"></category><category term="iptables"></category><category term="iproute"></category></entry></feed>