<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Michael Muratov</title><link>https://michaelmuratov.com/categories/ai/</link><description>Recent content on Michael Muratov</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><lastBuildDate>Sat, 11 Jul 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://michaelmuratov.com/categories/ai/index.xml" rel="self" type="application/rss+xml"/><item><title>LLM Tool Calling</title><link>https://michaelmuratov.com/blog/artifacts/guides/calculator-tool-call/</link><pubDate>Sat, 11 Jul 2026 00:00:00 +0000</pubDate><guid>https://michaelmuratov.com/blog/artifacts/guides/calculator-tool-call/</guid><description>&lt;h3 id="simple-math-is-actually-hard"&gt;Simple Math is actually Hard?&lt;/h3&gt;
&lt;p&gt;A very simple problem with large language models, which frequently gets highlighted, is their poor ability to accurately do simple grade school math. Something that would be easy for human beings, even just in their head, can be surprisingly difficult for these models to do reliably.&lt;/p&gt;
&lt;p&gt;There are many benchmarks for model performance on simple math problems, and their results are often disappointing. This isn&amp;rsquo;t that surprising, given that their training is based on next token prediction, which isn&amp;rsquo;t as well suited to number crunching like a calculator. Computers though, should be very good at number crunching. &lt;strong&gt;If we could just hook up LLMs to a calculator, we could fix this problem.&lt;/strong&gt;&lt;/p&gt;</description><content:encoded><![CDATA[<h3 id="simple-math-is-actually-hard">Simple Math is actually Hard?</h3>
<p>A very simple problem with large language models, which frequently gets highlighted, is their poor ability to accurately do simple grade school math. Something that would be easy for human beings, even just in their head, can be surprisingly difficult for these models to do reliably.</p>
<p>There are many benchmarks for model performance on simple math problems, and their results are often disappointing. This isn&rsquo;t that surprising, given that their training is based on next token prediction, which isn&rsquo;t as well suited to number crunching like a calculator. Computers though, should be very good at number crunching. <strong>If we could just hook up LLMs to a calculator, we could fix this problem.</strong></p>
<h3 id="llm-tool-calls">LLM Tool Calls</h3>
<p>If we can get the LLM to output something that <em>looks</em> like a function call, we can provide some parsing logic, actually execute the request deterministically, and provide the final answer back to the model. This way the model is only concerned with providing the right inputs, for which it can leverage its next token prediction, and then we can be certain that with the right inputs we will also be getting the right outputs.</p>
<h3 id="calculator-example">Calculator Example</h3>
<p>Let&rsquo;s write the simplest possible LLM tool call for a calculator. We can use Python for this.
A similar example is also provided by OpenAI&rsquo;s <a href="https://developers.openai.com/api/docs/guides/function-calling#function-tool-example">function calling</a> page.</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">  1</span><span class="cl"><span class="kn">from</span> <span class="nn">openai</span> <span class="kn">import</span> <span class="n">OpenAI</span>
</span></span><span class="line"><span class="ln">  2</span><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="ln">  3</span><span class="cl">
</span></span><span class="line"><span class="ln">  4</span><span class="cl"><span class="n">client</span> <span class="o">=</span> <span class="n">OpenAI</span><span class="p">(</span><span class="n">api_key</span><span class="o">=</span><span class="s2">&#34;your_api_key_here&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">  5</span><span class="cl">
</span></span><span class="line"><span class="ln">  6</span><span class="cl"><span class="c1"># 1. Define a list of callable tools for the model</span>
</span></span><span class="line"><span class="ln">  7</span><span class="cl"><span class="n">tools</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="ln">  8</span><span class="cl">    <span class="p">{</span>
</span></span><span class="line"><span class="ln">  9</span><span class="cl">        <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;function&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 10</span><span class="cl">        <span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="s2">&#34;calculate&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 11</span><span class="cl">        <span class="s2">&#34;description&#34;</span><span class="p">:</span> <span class="s2">&#34;Execute a simple arithmetic operation on two integers.&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 12</span><span class="cl">        <span class="s2">&#34;parameters&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 13</span><span class="cl">            <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;object&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 14</span><span class="cl">            <span class="s2">&#34;properties&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 15</span><span class="cl">                <span class="s2">&#34;first&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 16</span><span class="cl">                    <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;integer&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 17</span><span class="cl">                    <span class="s2">&#34;description&#34;</span><span class="p">:</span> <span class="s2">&#34;First number&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 18</span><span class="cl">                <span class="p">},</span>
</span></span><span class="line"><span class="ln"> 19</span><span class="cl">                <span class="s2">&#34;second&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 20</span><span class="cl">                    <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;integer&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 21</span><span class="cl">                    <span class="s2">&#34;description&#34;</span><span class="p">:</span> <span class="s2">&#34;Second number&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 22</span><span class="cl">                <span class="p">},</span>
</span></span><span class="line"><span class="ln"> 23</span><span class="cl">                <span class="s2">&#34;sign&#34;</span><span class="p">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 24</span><span class="cl">                    <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;string&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 25</span><span class="cl">                    <span class="s2">&#34;description&#34;</span><span class="p">:</span> <span class="s2">&#34;Symbol to execute the operation&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 26</span><span class="cl">                <span class="p">},</span>
</span></span><span class="line"><span class="ln"> 27</span><span class="cl">            <span class="p">},</span>
</span></span><span class="line"><span class="ln"> 28</span><span class="cl">            <span class="s2">&#34;required&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;first&#34;</span><span class="p">,</span> <span class="s2">&#34;second&#34;</span><span class="p">,</span> <span class="s2">&#34;sign&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="ln"> 29</span><span class="cl">        <span class="p">},</span>
</span></span><span class="line"><span class="ln"> 30</span><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="ln"> 31</span><span class="cl"><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 32</span><span class="cl">
</span></span><span class="line"><span class="ln"> 33</span><span class="cl"><span class="c1">#2. Define the function logic that will execute when the model calls the tool</span>
</span></span><span class="line"><span class="ln"> 34</span><span class="cl"><span class="k">def</span> <span class="nf">calculate</span><span class="p">(</span><span class="n">first</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">second</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">sign</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 35</span><span class="cl">    <span class="k">if</span> <span class="n">sign</span> <span class="o">==</span> <span class="s1">&#39;+&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 36</span><span class="cl">        <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">first</span> <span class="o">+</span> <span class="n">second</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 37</span><span class="cl">    <span class="k">elif</span> <span class="n">sign</span> <span class="o">==</span> <span class="s1">&#39;-&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 38</span><span class="cl">        <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">first</span> <span class="o">-</span> <span class="n">second</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 39</span><span class="cl">    <span class="k">elif</span> <span class="n">sign</span> <span class="o">==</span> <span class="s1">&#39;*&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 40</span><span class="cl">        <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">first</span> <span class="o">*</span> <span class="n">second</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 41</span><span class="cl">    <span class="k">elif</span> <span class="n">sign</span> <span class="o">==</span> <span class="s1">&#39;/&#39;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 42</span><span class="cl">        <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">first</span> <span class="o">/</span> <span class="n">second</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 43</span><span class="cl">    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 44</span><span class="cl">        <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;Unknown sign: </span><span class="si">{</span><span class="n">sign</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 45</span><span class="cl">
</span></span><span class="line"><span class="ln"> 46</span><span class="cl"><span class="c1"># Create a running input list we will add to over time</span>
</span></span><span class="line"><span class="ln"> 47</span><span class="cl"><span class="n">input_list</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="ln"> 48</span><span class="cl">    <span class="p">{</span><span class="s2">&#34;role&#34;</span><span class="p">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="s2">&#34;content&#34;</span><span class="p">:</span> <span class="s2">&#34;What is 221 * 645?&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 49</span><span class="cl"><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 50</span><span class="cl">
</span></span><span class="line"><span class="ln"> 51</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;User Question:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 52</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;What is 221 * 645?&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 53</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 54</span><span class="cl">
</span></span><span class="line"><span class="ln"> 55</span><span class="cl"><span class="c1"># 3. Prompt the model with tools defined</span>
</span></span><span class="line"><span class="ln"> 56</span><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">responses</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 57</span><span class="cl">    <span class="n">model</span><span class="o">=</span><span class="s2">&#34;gpt-4.1-mini&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 58</span><span class="cl">    <span class="n">tools</span><span class="o">=</span><span class="n">tools</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 59</span><span class="cl">    <span class="nb">input</span><span class="o">=</span><span class="n">input_list</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 60</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 61</span><span class="cl">
</span></span><span class="line"><span class="ln"> 62</span><span class="cl"><span class="c1"># Save function call outputs for subsequent requests</span>
</span></span><span class="line"><span class="ln"> 63</span><span class="cl"><span class="n">input_list</span> <span class="o">+=</span> <span class="n">response</span><span class="o">.</span><span class="n">output</span>
</span></span><span class="line"><span class="ln"> 64</span><span class="cl">
</span></span><span class="line"><span class="ln"> 65</span><span class="cl"><span class="c1"># 4. If the function call is present in the model&#39;s response, attempt to execute it</span>
</span></span><span class="line"><span class="ln"> 66</span><span class="cl"><span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">response</span><span class="o">.</span><span class="n">output</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 67</span><span class="cl">    <span class="k">if</span> <span class="n">item</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="s2">&#34;function_call&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 68</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Model Output:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 69</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 70</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;========LLM Invoked Function Call============&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 71</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;Function Call: </span><span class="si">{</span><span class="n">item</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 72</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;Arguments: </span><span class="si">{</span><span class="n">item</span><span class="o">.</span><span class="n">arguments</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 73</span><span class="cl">        <span class="k">if</span> <span class="n">item</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="s2">&#34;calculate&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 74</span><span class="cl">            <span class="c1"># 5. Execute the function logic for calculate</span>
</span></span><span class="line"><span class="ln"> 75</span><span class="cl">            <span class="n">first</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">arguments</span><span class="p">)[</span><span class="s2">&#34;first&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 76</span><span class="cl">            <span class="n">second</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">arguments</span><span class="p">)[</span><span class="s2">&#34;second&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 77</span><span class="cl">            <span class="n">sign</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">item</span><span class="o">.</span><span class="n">arguments</span><span class="p">)[</span><span class="s2">&#34;sign&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 78</span><span class="cl">            <span class="n">result</span> <span class="o">=</span> <span class="n">calculate</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">,</span> <span class="n">sign</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 79</span><span class="cl">            
</span></span><span class="line"><span class="ln"> 80</span><span class="cl">            <span class="c1"># 6. Provide function call results to the model</span>
</span></span><span class="line"><span class="ln"> 81</span><span class="cl">            <span class="n">input_list</span><span class="o">.</span><span class="n">append</span><span class="p">({</span>
</span></span><span class="line"><span class="ln"> 82</span><span class="cl">                <span class="s2">&#34;type&#34;</span><span class="p">:</span> <span class="s2">&#34;function_call_output&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 83</span><span class="cl">                <span class="s2">&#34;call_id&#34;</span><span class="p">:</span> <span class="n">item</span><span class="o">.</span><span class="n">call_id</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 84</span><span class="cl">                <span class="s2">&#34;output&#34;</span><span class="p">:</span> <span class="n">result</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 85</span><span class="cl">            <span class="p">})</span>
</span></span><span class="line"><span class="ln"> 86</span><span class="cl">
</span></span><span class="line"><span class="ln"> 87</span><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;Function Call Output: </span><span class="si">{</span><span class="n">result</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 88</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;=============================================&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 89</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 90</span><span class="cl">
</span></span><span class="line"><span class="ln"> 91</span><span class="cl"><span class="c1"># 5. Prompt the model again with function call outputs for final answer</span>
</span></span><span class="line"><span class="ln"> 92</span><span class="cl"><span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">responses</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 93</span><span class="cl">    <span class="n">model</span><span class="o">=</span><span class="s2">&#34;gpt-4.1-mini&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 94</span><span class="cl">    <span class="n">instructions</span><span class="o">=</span><span class="s2">&#34;Respond only with the full answer in plain text format.&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 95</span><span class="cl">    <span class="n">tools</span><span class="o">=</span><span class="n">tools</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 96</span><span class="cl">    <span class="nb">input</span><span class="o">=</span><span class="n">input_list</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 97</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 98</span><span class="cl">
</span></span><span class="line"><span class="ln"> 99</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;========LLM Recieved Call Answer============&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">100</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;Final output:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">101</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">output_text</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">102</span><span class="cl"><span class="nb">print</span><span class="p">(</span><span class="s2">&#34;============================================&#34;</span><span class="p">)</span></span></span></code></pre></div>


<link href="/css/admonitions.min.css" rel="stylesheet" />
  <div class="admonition important">
    <div class="admonition-header">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M256 512A256 256 0 1 0 256 0a256 256 0 1 0 0 512zm0-384c13.3 0 24 10.7 24 24l0 112c0 13.3-10.7 24-24 24s-24-10.7-24-24l0-112c0-13.3 10.7-24 24-24zM224 352a32 32 0 1 1 64 0 32 32 0 1 1 -64 0z"/></svg>
      <span>Important</span>
    </div>
      <div class="admonition-content">
        <p>Step 1 in the above code defined what calling the calculate function would require:</p>
<ol>
<li><strong>Type:</strong> function</li>
<li><strong>Name:</strong> calculate</li>
<li><strong>Parameters:</strong> first, second, sign</li>
</ol>
      </div>
  </div><p>When the model registers that calculate would be useful it sends a response to us that looks like this:</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="ln">1</span><span class="cl"><span class="o">[</span>ResponseFunctionToolCall<span class="o">(</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">  <span class="nv">type</span><span class="o">=</span><span class="s1">&#39;function_call&#39;</span>, 
</span></span><span class="line"><span class="ln">3</span><span class="cl">  <span class="nv">name</span><span class="o">=</span><span class="s1">&#39;calculate&#39;</span>,
</span></span><span class="line"><span class="ln">4</span><span class="cl">  <span class="nv">arguments</span><span class="o">=</span><span class="s1">&#39;{&#34;first&#34;:221,&#34;second&#34;:645,&#34;sign&#34;:&#34;*&#34;}&#39;</span>
</span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="o">)]</span></span></span></code></pre></div>
<p>At <strong>step 4</strong> in our code we check for any responses with type <strong>function</strong>. If we find one we parse out the function name. We use the provided paramters to run the appropriate function as a &ldquo;tool call&rdquo;. Then we simply provide the result of the function to our chat history for the model to observe.</p>
<p>The last part is simply asking the model to use the output from the function to answer the original question as a sentence which it can easily do.</p>
<p>Here is the full print output of the code block:</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="err">User</span> <span class="err">Question:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="err">What</span> <span class="err">is</span> <span class="mi">221</span> <span class="err">*</span> <span class="mi">645</span><span class="err">?</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="err">Model</span> <span class="err">Output:</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="err">ResponseFunctionToolCall(arguments=&#39;</span><span class="p">{</span><span class="nt">&#34;first&#34;</span><span class="p">:</span><span class="mi">221</span><span class="p">,</span><span class="nt">&#34;second&#34;</span><span class="p">:</span><span class="mi">645</span><span class="p">,</span><span class="nt">&#34;sign&#34;</span><span class="p">:</span><span class="s2">&#34;*&#34;</span><span class="p">}</span><span class="err">&#39;,</span> <span class="err">call_id=&#39;call_IA</span><span class="mi">3</span><span class="err">XFzdm</span><span class="mi">5</span><span class="err">YcamN</span><span class="mi">26</span><span class="err">Nd</span><span class="mi">7</span><span class="err">WeISX&#39;,</span> <span class="err">name=&#39;calculate&#39;,</span> <span class="err">type=&#39;function_call&#39;,</span> <span class="err">id=&#39;fc_</span><span class="mi">088</span><span class="err">a</span><span class="mi">460995856027006</span><span class="err">a</span><span class="mi">51</span><span class="err">ee</span><span class="mi">403</span><span class="err">ea</span><span class="mi">081929</span><span class="err">c</span><span class="mi">055</span><span class="err">bb</span><span class="mi">039</span><span class="err">b</span><span class="mi">6</span><span class="err">a</span><span class="mi">32</span><span class="err">c&#39;,</span> <span class="err">caller=None,</span> <span class="err">namespace=None,</span> <span class="err">status=&#39;completed&#39;)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="err">========LLM</span> <span class="err">Invoked</span> <span class="err">Function</span> <span class="err">Call============</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="err">Function</span> <span class="err">Call:</span> <span class="err">calculate</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="err">Arguments:</span> <span class="p">{</span><span class="nt">&#34;first&#34;</span><span class="p">:</span><span class="mi">221</span><span class="p">,</span><span class="nt">&#34;second&#34;</span><span class="p">:</span><span class="mi">645</span><span class="p">,</span><span class="nt">&#34;sign&#34;</span><span class="p">:</span><span class="s2">&#34;*&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="err">Function</span> <span class="err">Call</span> <span class="err">Output:</span> <span class="mi">142545</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="err">=============================================</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="err">========LLM</span> <span class="err">Recieved</span> <span class="err">Call</span> <span class="err">Answer============</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="err">Final</span> <span class="err">output:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="mi">221</span> <span class="err">multiplied</span> <span class="err">by</span> <span class="mi">645</span> <span class="err">is</span> <span class="mi">142</span><span class="err">,</span><span class="mi">545</span><span class="err">.</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="err">============================================</span></span></span></code></pre></div>


<link href="/css/admonitions.min.css" rel="stylesheet" />
  <div class="admonition info">
    <div class="admonition-header">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M256 512A256 256 0 1 0 256 0a256 256 0 1 0 0 512zM216 336l24 0 0-64-24 0c-13.3 0-24-10.7-24-24s10.7-24 24-24l48 0c13.3 0 24 10.7 24 24l0 88 8 0c13.3 0 24 10.7 24 24s-10.7 24-24 24l-80 0c-13.3 0-24-10.7-24-24s10.7-24 24-24zm40-208a32 32 0 1 1 0 64 32 32 0 1 1 0-64z"/></svg>
      <span>Info</span>
    </div>
      <div class="admonition-content">
        <p>By providing function calls to the model we can leverage our ability to execute functions reliably so long as the model is able to correctly identify when functions can be invoked and with the right arguments.</p>
      </div>
  </div><h3 id="generating-testing-values">Generating Testing Values</h3>
<p>In order to test how well a model can predict random arithmetic operations we have to first generate a few random values so that we can compare the results with and without tool calls.I picked two ways of generating values randomly:</p>
<ol>
<li>For a <strong>linear</strong> approach we pick a random magnitude from -5 to 5 as the exponent and a random value from -1 to 1 and we get a random value back of \([value * 10^{exp}]\)</li>
<li>For a <strong>gaussian</strong> approach we use a normal distribution with a set standard deviation. This ensures that our values are a bit closer to what we would consider normal, from around 100 to 0.001 with a low chance of outliers.</li>
</ol>
<p>Both distributions can be seen visualized below for 100 distinct values.</p>

<div><img src="value_distribution.png" 
		 class="img-preset img-preset-md" width="600" height="400">

	<figcaption style="margin:0px;" class="figure-caption">
		generate_graph.py
	</figcaption>
</div>

<p>Unfortunately the values which are generated for negative magnitudes of 10 are compressed into a very smal range inside -1 to 1.</p>
<p>It helps to plot the charts on a scale of magnitude versus their coefficients and draw the regions where 95% of the values (2 sigma) are found for each distribution. In the chart below we see that we get a much tighter range from the gassian distribution than we do from the linearly generated one. Based on the kind of values we want to test with we we can see if the model performs better or worse when the values are more or less spread out.</p>

<div><img src="scientific_distribution.png" 
		 class="img-preset img-preset-md" width="600" height="400">

	<figcaption style="margin:0px;" class="figure-caption">
		generate_graph.py
	</figcaption>
</div>

<p>We can also heatmap the values into a 2D grid to see where the values are concentrated. The heatmaps show the values for both the linear and gaussian distributions. This is a similar view to the sigma region chart above, but it shows the distribution in a more concentrated way.</p>

<div><img src="distribution_heatmaps.png" 
		 class="img-preset img-preset-md" width="600" height="400">

	<figcaption style="margin:0px;" class="figure-caption">
		generate_graph.py
	</figcaption>
</div>

<div style="display: flex; column-gap: 1rem; row-gap: 0.1rem; flex-wrap: wrap; justify-content: left;">
    
<a
  class="file-download"
  href="/blog/artifacts/guides/calculator-tool-call/log_distribution.py"
  download="log_distribution.py"
  aria-label="Download PY: log_distribution.py"
>
  <span class="file-download-icon" aria-hidden="true">
    <svg viewBox="0 0 24 24" focusable="false">
      <path d="M12 3v11m0 0 4-4m-4 4-4-4M5 18v2h14v-2" />
    </svg>
  </span>
  <span class="file-download-copy">
    <span class="file-download-label">Download PY</span>
    <span class="file-download-name">log_distribution.py</span>
  </span>
  <span class="file-download-action" aria-hidden="true">Click to download</span>
</a>
<a
  class="file-download"
  href="/blog/artifacts/guides/calculator-tool-call/plot_log_chart.py"
  download="plot_log_chart.py"
  aria-label="Download PY: plot_log_chart.py"
>
  <span class="file-download-icon" aria-hidden="true">
    <svg viewBox="0 0 24 24" focusable="false">
      <path d="M12 3v11m0 0 4-4m-4 4-4-4M5 18v2h14v-2" />
    </svg>
  </span>
  <span class="file-download-copy">
    <span class="file-download-label">Download PY</span>
    <span class="file-download-name">plot_log_chart.py</span>
  </span>
  <span class="file-download-action" aria-hidden="true">Click to download</span>
</a>
<a
  class="file-download"
  href="/blog/artifacts/guides/calculator-tool-call/plot_scientific_distribution.py"
  download="plot_scientific_distribution.py"
  aria-label="Download PY: plot_scientific_distribution.py"
>
  <span class="file-download-icon" aria-hidden="true">
    <svg viewBox="0 0 24 24" focusable="false">
      <path d="M12 3v11m0 0 4-4m-4 4-4-4M5 18v2h14v-2" />
    </svg>
  </span>
  <span class="file-download-copy">
    <span class="file-download-label">Download PY</span>
    <span class="file-download-name">plot_scientific_distribution.py</span>
  </span>
  <span class="file-download-action" aria-hidden="true">Click to download</span>
</a>
<a
  class="file-download"
  href="/blog/artifacts/guides/calculator-tool-call/plot_distribution_heatmaps.py"
  download="plot_distribution_heatmaps.py"
  aria-label="Download PY: plot_distribution_heatmaps.py"
>
  <span class="file-download-icon" aria-hidden="true">
    <svg viewBox="0 0 24 24" focusable="false">
      <path d="M12 3v11m0 0 4-4m-4 4-4-4M5 18v2h14v-2" />
    </svg>
  </span>
  <span class="file-download-copy">
    <span class="file-download-label">Download PY</span>
    <span class="file-download-name">plot_distribution_heatmaps.py</span>
  </span>
  <span class="file-download-action" aria-hidden="true">Click to download</span>
</a>

  </div>

<p>We&rsquo;ll use both of these distributions to generate random values for testing the model&rsquo;s ability to do simple math with and without tool calls. The next step is to run a series of tests using these generated values and compare the results.</p>
<h3 id="testing-the-model-without-tool-calls">Testing the Model without Tool Calls</h3>
<p>With the test data we have available we can pick two random values and a random operator and ask the model to calculate the result without using tool calls. We can then compare the model&rsquo;s output with the actual result of the operation and see how well it performs.</p>
]]></content:encoded></item><item><title>Tech Conference Resources</title><link>https://michaelmuratov.com/blog/artifacts/notes/tech-conference-resources/</link><pubDate>Thu, 18 Sep 2025 00:00:00 +0000</pubDate><guid>https://michaelmuratov.com/blog/artifacts/notes/tech-conference-resources/</guid><description>&lt;p&gt;A curated index of conference video archives by track. Channel links go to the full playlists page; year links go directly to a specific event.&lt;/p&gt;
&lt;p&gt;In 2026 the conferences worth your attention are the ones that answer three questions: where is the hardware going, what are attackers actually doing right now, and where are the hyperscalers taking the default infrastructure stack. The most useful sessions increasingly cut across all three. Most major US conferences offer virtual or on-demand passes at 40–50% of the in-person price; keynotes from AWS, Google, and Microsoft are typically free.&lt;/p&gt;</description><content:encoded><![CDATA[<p>A curated index of conference video archives by track. Channel links go to the full playlists page; year links go directly to a specific event.</p>
<p>In 2026 the conferences worth your attention are the ones that answer three questions: where is the hardware going, what are attackers actually doing right now, and where are the hyperscalers taking the default infrastructure stack. The most useful sessions increasingly cut across all three. Most major US conferences offer virtual or on-demand passes at 40–50% of the in-person price; keynotes from AWS, Google, and Microsoft are typically free.</p>
<hr>
<h3 id="ai--tech">AI &amp; Tech</h3>
<p>In 2026, watch GTC for the inference scaling and edge deployment roadmap. Whatever NVIDIA announces there shows up in production tooling 12–18 months later. At Build, I/O, and DevDay, the signal is agent framework maturity: how much of the scaffolding developers are hand-rolling today is getting absorbed into the platform. The academic conferences (NeurIPS, ICML, ICLR) are a longer time horizon and the papers published there become commercial products in two to three years, worth tracking if you want to get ahead of the next capability wave. Note that their year links go to paper listing pages, not video playlists; recorded talks are on SlidesLive or each conference&rsquo;s virtual portal.</p>
<table>
  <thead>
      <tr>
          <th>Channel</th>
          <th>Conference</th>
          <th>Years</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><a href="https://www.youtube.com/@AppleDeveloper/playlists">Apple</a></td>
          <td>WWDC</td>
          <td><a href="https://developer.apple.com/videos/">2020–2025</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@GoogleDevelopers/playlists">Google</a></td>
          <td>Google I/O</td>
          <td><a href="https://www.youtube.com/playlist?list=PLOU2XLYxmsIJEQRQDtuYKVUmxYvRfWN5m">2025</a> · <a href="https://www.youtube.com/playlist?list=PL590L5WQmH8doPo8OufXavO2Qu4ysZjyl">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/channel/UCrhJmfAGQ5K81XQ8_od1iTg/playlists">Microsoft</a></td>
          <td>Build</td>
          <td><a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHgFYS4DKbJ_xCYNE94ZLJjj">2025</a> · <a href="https://www.youtube.com/playlist?list=PLlrxD0HtieHghp_QCcQ2JiTXY3qO9oPDu">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@MetaDevelopers/playlists">Meta</a></td>
          <td>Connect</td>
          <td><a href="https://www.youtube.com/watch?v=EAZe8wDnAOA&amp;list=PLb0IAmt7-GS2cONiFVhtdKWEsyNkF6uUP">2025</a> · <a href="https://www.youtube.com/playlist?list=PLb0IAmt7-GS2bc5R9ZRFLAS78OXU3PJ6p">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@NVIDIADeveloper/playlists">NVIDIA</a></td>
          <td>GTC</td>
          <td><a href="https://www.youtube.com/playlist?list=PLZHnYvH1qtOYKcOvqqLtmfpLntjXc4LqT">2026</a> · <a href="https://www.youtube.com/playlist?list=PL5B692fm6--tCLg1NBPpsFGsIXOGbpVUa">2025</a> · <a href="https://www.youtube.com/playlist?list=PLn3ukorJv4vs_eN9cN8sXg0bq6CkX9lNy">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@OpenAI/playlists">OpenAI</a></td>
          <td>DevDay</td>
          <td><a href="https://www.youtube.com/playlist?list=PLOXw6I10VTv8-mTZk0v7oy1Bxfo3D2K5o">2025</a> · <a href="https://www.youtube.com/playlist?list=PLOXw6I10VTv_o0ZLpFu2IQyQOho1l-v7y">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@BlenderOfficial/featured">Blender</a></td>
          <td>Blender Conference</td>
          <td><a href="https://www.youtube.com/playlist?list=PLa1F2ddGya_8u-HEvmfCVuS_OImW8HaLd">2025</a> · <a href="https://www.youtube.com/playlist?list=PLa1F2ddGya_-Ymw4YlOjqrdQxiRMJql5x">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@NeurIPSConference/playlists">NeurIPS</a></td>
          <td>NeurIPS</td>
          <td><a href="https://slideslive.com/neurips-2025">2025</a> · <a href="https://slideslive.com/neurips-2024">2024</a> · <a href="https://slideslive.com/neurips-2023">2023</a> · <a href="https://slideslive.com/neurips-2022">2022</a> · <a href="https://slideslive.com/neurips-2021">2021</a> · <a href="https://slideslive.com/neurips-2020">2020</a></td>
      </tr>
      <tr>
          <td><a href="https://icml.cc">ICML</a></td>
          <td>ICML</td>
          <td><a href="https://icml.cc/virtual/2025/papers.html">2025</a> · <a href="https://icml.cc/virtual/2024/papers.html">2024</a> · <a href="https://icml.cc/virtual/2023/papers.html">2023</a> · <a href="https://icml.cc/virtual/2022/papers.html">2022</a> · <a href="https://icml.cc/virtual/2021/papers.html">2021</a> · <a href="https://icml.cc/virtual/2020/papers.html">2020</a></td>
      </tr>
      <tr>
          <td><a href="https://iclr.cc">ICLR</a></td>
          <td>ICLR</td>
          <td><a href="https://iclr.cc/virtual/2025/papers.html">2025</a> · <a href="https://iclr.cc/virtual/2024/papers.html">2024</a> · <a href="https://iclr.cc/virtual/2023/papers.html">2023</a> · <a href="https://iclr.cc/virtual/2022/papers.html">2022</a> · <a href="https://iclr.cc/virtual/2021/papers.html">2021</a> · <a href="https://iclr.cc/virtual/2020/papers.html">2020</a></td>
      </tr>
  </tbody>
</table>
<hr>
<h3 id="security">Security</h3>
<p>The sessions most worth your time in 2026 are about what AI is enabling on the offensive side: automated vulnerability discovery, LLM-driven social engineering at scale, and ML-based identity spoofing that bypasses static MFA. The Black Hat Arsenal track is the best early-warning signal showcasing open-source tools that will be repurposed against production systems. If you can&rsquo;t attend in person, the On-Demand pass (~$1,300–$1,800) covers all recorded Briefings about a week post-event. DEF CON has a youtube playlist which is populated after the conference.</p>
<table>
  <thead>
      <tr>
          <th>Channel</th>
          <th>Conference</th>
          <th>Years</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><a href="https://www.youtube.com/@RSAConference/playlists">RSA Conference</a></td>
          <td>RSA Conference</td>
          <td><a href="https://www.youtube.com/playlist?list=PLeUGLKUYzh_j6tll9I__IX6pg4hDZIvSD">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@AWSEventsChannel">AWS</a></td>
          <td>AWS re:Inforce</td>
          <td><a href="https://www.youtube.com/playlist?list=PL2yQDdvlhXf9XkXzO5bXvtMaio6gAcdmN">2025</a> · <a href="https://www.youtube.com/playlist?list=PL2yQDdvlhXf-a63_o1-mAya81ZwuRkmAu">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@BlackHatOfficialYT/playlists">Black Hat</a></td>
          <td>Black Hat USA</td>
          <td><a href="https://www.youtube.com/playlist?list=PLH15HpR5qRsXtV3t9TlkRZBmIfcIC6THE">2025</a> · <a href="https://www.youtube.com/playlist?list=PLH15HpR5qRsUiLYPNSylDvlskvS_RSzee">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@BlackHatOfficialYT/playlists">Black Hat</a></td>
          <td>Black Hat Asia</td>
          <td><a href="https://www.youtube.com/playlist?list=PLH15HpR5qRsWXApxW0vAsRduq9xa3ifCG">2025</a> · <a href="https://www.youtube.com/watch?v=H-7ZNrpsV50&amp;list=PLH15HpR5qRsWJ70J405J7z1GUYPeCBpXy">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@DEFCONConference/videos">DEF CON</a></td>
          <td>DEF CON</td>
          <td><a href="https://www.youtube.com/playlist?list=PL9fPq3eQfaaBB9HANwjaTlKFuVhRxkdz2">2025</a> · <a href="https://www.youtube.com/playlist?list=PLStO1VqVBvmGyghpuCaosPGtrwVy5Fbqi">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@UsenixOrg/playlists">USENIX Security</a></td>
          <td>USENIX Security</td>
          <td><a href="https://www.youtube.com/playlist?list=PLbRoZ5Rrl5ldYBeWL_T1Ei3uefAy1VtNB">2025</a> · <a href="https://www.youtube.com/playlist?list=PLbRoZ5Rrl5ldQ2K_dpmPKHEyRgyf5JSxd">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@SANSInstitute/playlists">SANS Institute</a></td>
          <td>SANS AI Cybersecurity Summit</td>
          <td><a href="https://www.sans.org/cyber-security-training-events/ai-summit-2026">2026</a></td>
      </tr>
  </tbody>
</table>
<hr>
<h3 id="engineering--cloud">Engineering &amp; Cloud</h3>
<p>In 2026 the hyperscaler keynotes are where you calibrate your architecture assumptions, specifically what AWS Bedrock, Google Gemini integrations, and Azure AI are absorbing that developers currently build themselves. The security implications follow directly: whatever gets built into the platform becomes the new baseline that attackers target. The spring window (late April through late May) clusters Google Cloud Next, the SANS AI Summit, and KCD Toronto within roughly six weeks.</p>
<table>
  <thead>
      <tr>
          <th>Channel</th>
          <th>Conference</th>
          <th>Years</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><a href="https://www.youtube.com/@AWSEventsChannel/playlists">AWS</a></td>
          <td>re:Invent</td>
          <td><a href="https://www.youtube.com/playlist?list=PL2yQDdvlhXf9gdFFBcDPUHAJS7kkIkIet">2025</a> · <a href="https://www.youtube.com/playlist?list=PL2yQDdvlhXf_ZsP25dGLTNbrVSphM2JDl">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@googlecloudtech/playlists">Google Cloud</a></td>
          <td>Google Cloud Next</td>
          <td><a href="https://www.youtube.com/playlist?list=PLIivd44G4vL3f30B0dK_Pz1z15J6Yh48H">2025</a> · <a href="https://www.youtube.com/playlist?list=PLIivdWyY5sqLHU6fh9ozZ7mxsj7GcPjRF">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@GitHub/playlists">GitHub</a></td>
          <td>Universe</td>
          <td><a href="https://www.youtube.com/playlist?list=PL0lo9MOBetEFKNlPHNouEmVeYeyoyGTXC">2025</a> · <a href="https://www.youtube.com/playlist?list=PL0lo9MOBetEF_de7yKAWpnMkTsKH6aJ4P">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@DockerInc">Docker</a></td>
          <td>DockerCon</td>
          <td><a href="https://www.youtube.com/playlist?list=PLkA60AVN3hh_t73mQG7RIvvMRTpm1ompt">2023</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@Kubernetesio/playlists">Kubernetes</a></td>
          <td>KubeCon + CloudNativeCon</td>
          <td><a href="https://www.youtube.com/playlist?list=PLj6h78yzYM2Of6psbKjeZxZIHjycSYoYF">2025</a> · <a href="https://www.youtube.com/playlist?list=PLj6h78yzYM2Pw4mRw4S-1p_xLARMqPkA7">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@GOTO-/playlists">GoTo Conference</a></td>
          <td>Top 10</td>
          <td><a href="https://www.youtube.com/playlist?list=PLEx5khR4g7PJ-PseKvyEdYIg3X6xv045b">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@PyConUS/playlists">PyCon US</a></td>
          <td>PyCon US</td>
          <td><a href="https://www.youtube.com/playlist?list=PL2Uw4_HvXqvb98mQjN0-rYQjdDxJ_hcrs">2025</a> · <a href="https://www.youtube.com/playlist?list=PL2Uw4_HvXqvYhjub9bw4uDAmNtprgAvlJ">2024</a> · <a href="https://www.youtube.com/playlist?list=PL2Uw4_HvXqvY2zhJ9AMUa_Z6dtMGF3gtb">2023</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/channel/UCrhJmfAGQ5K81XQ8_od1iTg/playlists">Microsoft</a></td>
          <td>Ignite</td>
          <td><a href="https://www.youtube.com/playlist?list=PLQXpv_NQsPIDXiR9PcpggZ34mzko_-12C">2025</a> · <a href="https://www.youtube.com/playlist?list=PLQXpv_NQsPID0sNvENCDMADnd1M5aOfv4">2024</a></td>
      </tr>
      <tr>
          <td><a href="https://www.youtube.com/@Databricks/playlists">Databricks</a></td>
          <td>Data + AI Summit</td>
          <td><a href="https://www.youtube.com/playlist?list=PLTPXxbhUt-YWjDg318nmSxRqTgZHIghTl">2025</a> · <a href="https://www.youtube.com/playlist?list=PLTPXxbhUt-YWjq4SDFS3_x0-x5KVT5GkQ">2024</a></td>
      </tr>
  </tbody>
</table>
<hr>
<h3 id="canada">Canada</h3>
<p>Toronto-accessible events where the value is physical attendance without international travel overhead. SecTor is the largest local conference for security professionals and a good place to learn about how the global threat landscape maps onto the Canadian companies. NorthSec is where you go to actually build offensive skills: the 2026 curriculum includes dedicated AI security training tracks covering practical attack and defense with ML. Video archives for Canadian events are inconsistent; check the channel links after each event.</p>
<table>
  <thead>
      <tr>
          <th>Channel</th>
          <th>Conference</th>
          <th>Years</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><a href="https://www.youtube.com/@BlackHatOfficialYT/playlists">Black Hat</a></td>
          <td>SecTor</td>
          <td><a href="http://www.2024.video.sector.ca/">2024</a> · <a href="http://www.2023.video.sector.ca/">2023</a> · <a href="http://www.2022.video.sector.ca/">2022</a> · <a href="http://www.2021.video.sector.ca/">2021</a> · <a href="http://www.2020.video.sector.ca/">2020</a></td>
      </tr>
      <tr>
          <td><a href="https://nsec.io/">NorthSec</a></td>
          <td>NorthSec</td>
          <td><a href="https://www.youtube.com/playlist?list=PLuUtcRxSUZUrW9scJZqhbiuTBwZBJ-Qic">2025</a></td>
      </tr>
      <tr>
          <td><a href="https://www.cncf.io/">CNCF</a></td>
          <td>KCD Toronto</td>
          <td><a href="https://www.youtube.com/playlist?list=PLj6h78yzYM2MXCOWSN9CqqID6OOvF7wxL">2026</a></td>
      </tr>
  </tbody>
</table>
]]></content:encoded></item><item><title>AI Tools I'm Using Right Now</title><link>https://michaelmuratov.com/blog/artifacts/notes/ai-tools-using-now/</link><pubDate>Fri, 16 May 2025 00:00:00 +0000</pubDate><guid>https://michaelmuratov.com/blog/artifacts/notes/ai-tools-using-now/</guid><description>&lt;h3 id="chatgpt"&gt;ChatGPT&lt;/h3&gt;
&lt;link href="https://michaelmuratov.com/css/admonitions.min.css" rel="stylesheet" /&gt;
&lt;div class="admonition note"&gt;
&lt;div class="admonition-header"&gt;
&lt;svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 576 512"&gt;&lt;path d="M0 64C0 28.7 28.7 0 64 0L224 0l0 128c0 17.7 14.3 32 32 32l128 0 0 125.7-86.8 86.8c-10.3 10.3-17.5 23.1-21 37.2l-18.7 74.9c-2.3 9.2-1.8 18.8 1.3 27.5L64 512c-35.3 0-64-28.7-64-64L0 64zm384 64l-128 0L256 0 384 128zM549.8 235.7l14.4 14.4c15.6 15.6 15.6 40.9 0 56.6l-29.4 29.4-71-71 29.4-29.4c15.6-15.6 40.9-15.6 56.6 0zM311.9 417L441.1 287.8l71 71L382.9 487.9c-4.1 4.1-9.2 7-14.9 8.4l-60.1 15c-5.5 1.4-11.2-.2-15.2-4.2s-5.6-9.7-4.2-15.2l15-60.1c1.4-5.6 4.3-10.8 8.4-14.9z"/&gt;&lt;/svg&gt;
&lt;span&gt;Context&lt;/span&gt;
&lt;/div&gt;
&lt;div class="admonition-content"&gt;
&lt;p&gt;I&amp;rsquo;ve been using ChatGPT since it came out in &lt;code&gt;late 2022&lt;/code&gt;&lt;/p&gt;</description><content:encoded><![CDATA[<h3 id="chatgpt">ChatGPT</h3>


<link href="/css/admonitions.min.css" rel="stylesheet" />
  <div class="admonition note">
    <div class="admonition-header">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 576 512"><path d="M0 64C0 28.7 28.7 0 64 0L224 0l0 128c0 17.7 14.3 32 32 32l128 0 0 125.7-86.8 86.8c-10.3 10.3-17.5 23.1-21 37.2l-18.7 74.9c-2.3 9.2-1.8 18.8 1.3 27.5L64 512c-35.3 0-64-28.7-64-64L0 64zm384 64l-128 0L256 0 384 128zM549.8 235.7l14.4 14.4c15.6 15.6 15.6 40.9 0 56.6l-29.4 29.4-71-71 29.4-29.4c15.6-15.6 40.9-15.6 56.6 0zM311.9 417L441.1 287.8l71 71L382.9 487.9c-4.1 4.1-9.2 7-14.9 8.4l-60.1 15c-5.5 1.4-11.2-.2-15.2-4.2s-5.6-9.7-4.2-15.2l15-60.1c1.4-5.6 4.3-10.8 8.4-14.9z"/></svg>
      <span>Context</span>
    </div>
      <div class="admonition-content">
        <p>I&rsquo;ve been using ChatGPT since it came out in <code>late 2022</code></p>
      </div>
  </div><p>I first caught the wave of excitement over the limited release of GPT3.5 (aka ChatGPT), a language model that was surprisingly good at generating text. Back then the use case was simple, I gave it ridiculous story premises to see what it could generate. I had a great time reading these stories with friends and we&rsquo;d all laugh at how funny and interesting they were. I had very little foresight as to how much things would change. I thought it would be a short term beta, with limited release and invite only access. I had no sense of how permanent it would become.</p>
<p>Since then my usage of ChatGPT has evolved over time, influenced by the gradual improvement of the models, and the exploration of their limits. I love it for surfacing up new ideas, writing short code snippets, and exploring hypotheticals. The versatility of ChatGPT is what keeps me coming back to it as the go-to chatbot. As long as OpenAI keeps innovating on features that enhance the already powerful language model, I expect I will keep coming back to it for most of my needs.</p>
<h3 id="copilot">Copilot</h3>


<link href="/css/admonitions.min.css" rel="stylesheet" />
  <div class="admonition note">
    <div class="admonition-header">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 576 512"><path d="M0 64C0 28.7 28.7 0 64 0L224 0l0 128c0 17.7 14.3 32 32 32l128 0 0 125.7-86.8 86.8c-10.3 10.3-17.5 23.1-21 37.2l-18.7 74.9c-2.3 9.2-1.8 18.8 1.3 27.5L64 512c-35.3 0-64-28.7-64-64L0 64zm384 64l-128 0L256 0 384 128zM549.8 235.7l14.4 14.4c15.6 15.6 15.6 40.9 0 56.6l-29.4 29.4-71-71 29.4-29.4c15.6-15.6 40.9-15.6 56.6 0zM311.9 417L441.1 287.8l71 71L382.9 487.9c-4.1 4.1-9.2 7-14.9 8.4l-60.1 15c-5.5 1.4-11.2-.2-15.2-4.2s-5.6-9.7-4.2-15.2l15-60.1c1.4-5.6 4.3-10.8 8.4-14.9z"/></svg>
      <span>Context</span>
    </div>
      <div class="admonition-content">
        <p>Copilot was the first code specific language model I tried and I&rsquo;ve been using it ever since</p>
      </div>
  </div><p>Copilot works great as a basic autocomplete to help bubble up related functions that I can choose to add in my project. For planning out a software project I use other language models with reasoning advanced functions.</p>
]]></content:encoded></item><item><title>Toy Car Reinforcement Learning</title><link>https://michaelmuratov.com/blog/artifacts/builds/toy-car-ml/</link><pubDate>Sun, 11 Aug 2019 00:00:00 +0000</pubDate><guid>https://michaelmuratov.com/blog/artifacts/builds/toy-car-ml/</guid><description>&lt;h3 id="introduction"&gt;Introduction&lt;/h3&gt;
&lt;p&gt;As a team of three undergraduate students, we were tasked with building a concept self driving car in our winter semester. Not having much experience in data science or machine learning meant that the production process was a mess, although looking back we probably could have made the same amount of progress in a fraction of the time.&lt;/p&gt;
&lt;p&gt;This article is for those who are interested in what it takes to build a functioning self driving car in record time, on a budget, without the headaches that we had to go through! I don’t go too much in depth on all the details but there are a few snippets of code to spice things up. Also if you are interested in the code you can help yourself to our Github page, &lt;a href="https://github.com/RoboticsCourse"&gt;https://github.com/RoboticsCourse&lt;/a&gt;, but be warned that it’s not documented and was not created for general use.&lt;/p&gt;</description><content:encoded><![CDATA[<h3 id="introduction">Introduction</h3>
<p>As a team of three undergraduate students, we were tasked with building a concept self driving car in our winter semester. Not having much experience in data science or machine learning meant that the production process was a mess, although looking back we probably could have made the same amount of progress in a fraction of the time.</p>
<p>This article is for those who are interested in what it takes to build a functioning self driving car in record time, on a budget, without the headaches that we had to go through! I don’t go too much in depth on all the details but there are a few snippets of code to spice things up. Also if you are interested in the code you can help yourself to our Github page, <a href="https://github.com/RoboticsCourse">https://github.com/RoboticsCourse</a>, but be warned that it’s not documented and was not created for general use.</p>
<h3 id="what-well-need">What We’ll Need:</h3>
<ul>
<li><a href="https://www.amazon.com/dp/B09YTG1LFD/ref=sspa_dk_detail_3?psc=1&amp;pd_rd_i=B09YTG1LFD&amp;pd_rd_w=3pESF&amp;content-id=amzn1.sym.386c274b-4bfe-4421-9052-a1a56db557ab&amp;pf_rd_p=386c274b-4bfe-4421-9052-a1a56db557ab&amp;pf_rd_r=6RPTHKRZEY7TXSRJBBEG&amp;pd_rd_wg=7xuuP&amp;pd_rd_r=24695953-241e-4d3d-9f00-f71d2b63e104&amp;s=toys-and-games&amp;sp_csd=d2lkZ2V0TmFtZT1zcF9kZXRhaWxfdGhlbWF0aWM">Remote Control/Demo Car</a></li>
<li><a href="https://www.amazon.com/gp/product/B008GRTSV6/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=B008GRTSV6&amp;linkCode=as2&amp;tag=michaelmurato-20&amp;linkId=5f8175e1eb6844326ad63db3658f1215">Arduino Uno board</a></li>
<li><a href="https://www.amazon.com/gp/product/B00PUTH3B0/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=B00PUTH3B0&amp;linkCode=as2&amp;tag=michaelmurato-20&amp;linkId=11c76b435eb0e749b7ecf059377394c2">Motor Shield</a></li>
<li><a href="https://www.amazon.com/gp/product/B01HN72K14/ref=as_li_tl?ie=UTF8&amp;camp=1789&amp;creative=9325&amp;creativeASIN=B01HN72K14&amp;linkCode=as2&amp;tag=michaelmurato-20&amp;linkId=2419b3f00601a89ef9ec37ea1cad1ecf">Bluetooth BLE Module</a></li>
<li>2 Mobile Devices</li>
</ul>
<h3 id="cars-are-meant-for-driving">Cars are meant for driving</h3>
<p>First of all, we wanted to finely control the motion of the car, and the best way to do that was to throw away the controller that came with the original car and instead use an Arduino board. Since they’re programmable and compatible with a whole suite of additional modules which we could buy online for cheap, it was amazing for prototyping. The board we used was the Arduino UNO Rev3 which you can get for $20.</p>
<p>Now to actually be able to control the car via the Arduino board we had to hook it up to a Motor Shield which is a useful little add-on that gives you the ability to power a variety of motors from your Arduino board with just a few lines of code. The snippet bellow shows a simple implementation.</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-cpp" data-lang="cpp"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="kt">void</span> <span class="n">Navigation</span><span class="o">::</span><span class="n">goForward</span><span class="p">(</span><span class="kt">int</span> <span class="n">speed</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">    <span class="n">frontMotor</span><span class="o">-&gt;</span><span class="n">setSpeed</span><span class="p">(</span><span class="n">speed</span><span class="p">);</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="n">frontMotor</span><span class="o">-&gt;</span><span class="n">run</span><span class="p">(</span><span class="n">FORWARD</span><span class="p">);</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="n">rearMotor</span><span class="o">-&gt;</span><span class="n">setSpeed</span><span class="p">(</span><span class="n">speed</span><span class="p">);</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="n">rearMotor</span><span class="o">-&gt;</span><span class="n">run</span><span class="p">(</span><span class="n">FORWARD</span><span class="p">);</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="kt">void</span> <span class="n">Navigation</span><span class="o">::</span><span class="n">goBackward</span><span class="p">(</span><span class="kt">int</span> <span class="n">speed</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="n">frontMotor</span><span class="o">-&gt;</span><span class="n">setSpeed</span><span class="p">(</span><span class="n">speed</span><span class="p">);</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="n">frontMotor</span><span class="o">-&gt;</span><span class="n">run</span><span class="p">(</span><span class="n">BACKWARD</span><span class="p">);</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="n">rearMotor</span><span class="o">-&gt;</span><span class="n">setSpeed</span><span class="p">(</span><span class="n">speed</span><span class="p">);</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="n">rearMotor</span><span class="o">-&gt;</span><span class="n">run</span><span class="p">(</span><span class="n">BACKWARD</span><span class="p">);</span>   
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">
</span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="kt">void</span> <span class="n">Navigation</span><span class="o">::</span><span class="n">SteerSpeed</span><span class="p">(</span><span class="kt">int</span> <span class="n">speed</span><span class="p">){</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">  <span class="k">if</span><span class="p">(</span><span class="n">speed</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">)</span> <span class="n">turnMotor</span><span class="o">-&gt;</span><span class="n">run</span><span class="p">(</span><span class="n">FORWARD</span><span class="p">);</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">  <span class="k">else</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="n">speed</span> <span class="o">*=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">    <span class="n">turnMotor</span><span class="o">-&gt;</span><span class="n">run</span><span class="p">(</span><span class="n">BACKWARD</span><span class="p">);</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">  <span class="n">turnMotor</span><span class="o">-&gt;</span><span class="n">setSpeed</span><span class="p">(</span><span class="n">speed</span><span class="p">);</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="p">}</span></span></span></code></pre></div>
<p>An important implementation challenge was that there is no way to specify distances that the motor will be doing useful work over but instead you must pass in the speed the motor will turn at and the time interval. This means that deepening on the amount of power supplied by the motor, you are responsible for fine tuning these measurements.</p>
<p>Ours ran three DC Motors which were well supported by our Motor Shield. This setup guide really helped us set up and run this configuration.</p>

<div><img src="arduino.webp" 
		 class="img-preset img-preset-md" width="600" height="400">
</div>

<p>Now that the Arduino + Motor Shield were all hooked up, and the motors were buzzing we gained the ability to control the car through its new small Arduino brain. However this early in the project it could only follow pre-programmed instructions in a loop and it was time to test what we’ve built.</p>
<h3 id="manual-control">Manual Control</h3>
<p>The next step was to hook up a way to control the car manually, which turned out to be really useful in the training phase. A really handy module available in the Arduino arsenal is the Bluetooth Low Energy chip (BLE). The setup is quick and there’s available example source code for mobile devices to BLE connection so it only took us some minor tweaking to get the board connected to one of our phones.</p>
<p>We used an Android phone to interface with the chip, the the help of this <a href="https://github.com/googlearchive/android-BluetoothLeGatt">BLE Android project</a>. That sample code is an example of Bluetooth text messaging which is very similar to what we’ll be doing to communicate with the car, in fact our implementation will be even simpler.</p>
<p>All it takes is to scan for local Bluetooth devices and connect to the chip, after which we can start sending and receiving messages in a few lines of code! This gave us the freedom to go wild with the input type for the car controller, with my preference being a simple joystick. By sending messages such as S10 (Steer Left at speed 10) or F -100 (Go backwards at speed 100), with S representing the steering motor and F representing the front and back motors, we could encode directions for the car in just a few bytes.</p>

<div><img src="android.webp" 
		 class="img-preset img-preset-md" width="600" height="400">
</div>

<p>The latency between the controller and the car proved to be nonexistent and even worked for up to 10m away with no issues. At this point the car was controllable in the same way it was before, if not better. The next step was to actually make car drive on its own and after a few failed attempts at coming up with logic for the car, we turned to neural networks.</p>
<h3 id="machine-learning-needs-data">Machine Learning Needs Data</h3>
<p>Since with neural networks we really can’t get anywhere without gathering a large data set, it’s time to roll out the car and gather sweet data. Like any explorer venturing out into the unknown, our car will want to record all of its experiences for future self reflection. This means that we’ll need one mobile device to drive the car (from the implementation above), and another device with a camera strapped to the front of the car.</p>

<div><img src="train_process.webp" 
		 class="img-preset img-preset-md" width="150" height="100">

	<figcaption style="margin:0px;" class="figure-caption">
		Mobile devices in red
	</figcaption>
</div>

<p>Because our budget was small we strapped a second phone to the car to record video as the car drove around. The input data we could collect was what the camera on the car can see and the logs left by the controller app as a human pilots the car with the other phone. As it turns out, this setup we had now, was the bare minimum required to train a neural network, featuring the input set and a label set.</p>
<p>The input set was the data captured by the camera on the car, and the label set was the input of the human driver piloting the remote car. Now how does that second set work as a label set exactly?</p>
<h3 id="imitation-learning">Imitation Learning</h3>
<p>Do you remember the very first steps you took as a child? Chances are you did not. Even before conscious thought develops in the brain, its capacity to imitate the actions of other people are baked into the wiring of our brains and it allows for learning at an astonishing rate. Our car can’t “walk” yet but just like in the case of the baby, it’s our job to give it an example to follow.</p>
<p>By having a method of controlling the car and logging the input from the human, our car can learn what input it should supply to itself over time and become as good as the pilot.</p>

<div><img src="test_process.webp" 
		 class="img-preset img-preset-md" width="150" height="100">

	<figcaption style="margin:0px;" class="figure-caption">
		Celebrate while you still can, human
	</figcaption>
</div>

<h3 id="gathering-the-data">Gathering the Data</h3>
<p>We created a diagram of extreme values for the joystick. The first coordinate value being the degree of steering (x axis) and the second value being the amount of motion (y axis). Having a line of (0,0) cut through the middle of the joystick allowed for a gradient of speed between moving full force forward and moving full force backwards, reducing the wear and tear on our motors, and with maximum steering at 45 degree angles.</p>

<div><img src="input.webp" 
		 class="img-preset img-preset-md" width="600" height="400">
</div>

<p>Below is the trigonometry that constrains the input joystick to a circle</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="kt">float</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">event</span><span class="p">.</span><span class="na">getX</span><span class="p">()</span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="kt">float</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">event</span><span class="p">.</span><span class="na">getY</span><span class="p">()</span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="kt">double</span><span class="w"> </span><span class="n">distance</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">sqrt</span><span class="p">(</span><span class="n">Math</span><span class="p">.</span><span class="na">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">2</span><span class="p">)</span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">pow</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">2</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="kt">double</span><span class="w"> </span><span class="n">angle</span><span class="w">  </span><span class="o">=</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">atan2</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="n">distance</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="p">){</span><span class="w"> </span><span class="c1">//if outside the joystick</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">    </span><span class="n">cursorX</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="p">)</span><span class="o">*</span><span class="n">Math</span><span class="p">.</span><span class="na">sin</span><span class="p">(</span><span class="n">angle</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w">    </span><span class="n">cursorY</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="w"> </span><span class="p">((</span><span class="n">circle</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="p">)</span><span class="o">*</span><span class="n">Math</span><span class="p">.</span><span class="na">cos</span><span class="p">(</span><span class="n">angle</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w">    </span><span class="n">cursorX</span><span class="o">+=</span><span class="n">circle</span><span class="p">.</span><span class="na">getX</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="n">control</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w">    </span><span class="n">cursorY</span><span class="o">+=</span><span class="n">circle</span><span class="p">.</span><span class="na">getY</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="n">control</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="k">else</span><span class="p">{</span><span class="w"> </span><span class="c1">//if inside the joystick</span><span class="w">
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="w">    </span><span class="n">cursorX</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">x</span><span class="o">+</span><span class="n">circle</span><span class="p">.</span><span class="na">getX</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="n">control</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="w">    </span><span class="n">cursorY</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">y</span><span class="o">+</span><span class="n">circle</span><span class="p">.</span><span class="na">getY</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="p">(</span><span class="n">control</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="w"> </span><span class="o">&gt;&gt;</span><span class="w"> </span><span class="n">1</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="n">control</span><span class="p">.</span><span class="na">setX</span><span class="p">(</span><span class="n">cursorX</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="n">control</span><span class="p">.</span><span class="na">setY</span><span class="p">(</span><span class="n">cursorY</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">21</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="kd">final</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">vector_X</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">)(</span><span class="n">circle</span><span class="p">.</span><span class="na">getX</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">control</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">cursorX</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">255</span><span class="o">/</span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getWidth</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">23</span><span class="cl"><span class="kd">final</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">vector_Y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">)(</span><span class="n">circle</span><span class="p">.</span><span class="na">getY</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">circle</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">control</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">cursorY</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">255</span><span class="o">/</span><span class="p">(</span><span class="n">circle</span><span class="p">.</span><span class="na">getHeight</span><span class="p">()</span><span class="o">/</span><span class="n">2</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">25</span><span class="cl"><span class="kt">int</span><span class="w"> </span><span class="n">final_distance</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">dist</span><span class="o">*</span><span class="n">distance</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">26</span><span class="cl"><span class="k">if</span><span class="p">(</span><span class="n">vector_X</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">0</span><span class="p">){</span><span class="w">
</span></span></span><span class="line"><span class="ln">27</span><span class="cl"><span class="w">      </span><span class="n">final_distance</span><span class="w"> </span><span class="o">*=</span><span class="w"> </span><span class="o">-</span><span class="n">1</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln">28</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">29</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">30</span><span class="cl"><span class="n">distance</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">sqrt</span><span class="p">(</span><span class="n">Math</span><span class="p">.</span><span class="na">pow</span><span class="p">(</span><span class="n">vector_Y</span><span class="p">,</span><span class="w"> </span><span class="n">2</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">pow</span><span class="p">(</span><span class="n">vector_X</span><span class="p">,</span><span class="w"> </span><span class="n">2</span><span class="p">));</span><span class="w">
</span></span></span><span class="line"><span class="ln">31</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">32</span><span class="cl"><span class="c1">//get scaled values between -255 and 255 for X and between -150 and 150 for Y</span><span class="w">
</span></span></span><span class="line"><span class="ln">33</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">34</span><span class="cl"><span class="kt">float</span><span class="w"> </span><span class="n">dist</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">0</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln">35</span><span class="cl"><span class="k">if</span><span class="p">(</span><span class="n">Math</span><span class="p">.</span><span class="na">abs</span><span class="p">(</span><span class="n">vector_X</span><span class="p">)</span><span class="w"> </span><span class="o">&gt;=</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">abs</span><span class="p">(</span><span class="n">vector_Y</span><span class="p">)){</span><span class="w">
</span></span></span><span class="line"><span class="ln">36</span><span class="cl"><span class="w">    </span><span class="k">if</span><span class="p">(</span><span class="n">vector_X</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">0</span><span class="p">){</span><span class="w">
</span></span></span><span class="line"><span class="ln">37</span><span class="cl"><span class="w">        </span><span class="n">dist</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">abs</span><span class="p">(</span><span class="n">vector_Y</span><span class="p">)</span><span class="o">/</span><span class="n">Math</span><span class="p">.</span><span class="na">abs</span><span class="p">(</span><span class="n">vector_X</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">38</span><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">39</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">40</span><span class="cl"><span class="k">else</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="ln">41</span><span class="cl"><span class="w">    </span><span class="k">if</span><span class="p">(</span><span class="n">vector_Y</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="n">0</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="ln">42</span><span class="cl"><span class="w">        </span><span class="n">dist</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="w"> </span><span class="n">Math</span><span class="p">.</span><span class="na">abs</span><span class="p">(</span><span class="n">vector_X</span><span class="p">)</span><span class="o">/</span><span class="n">Math</span><span class="p">.</span><span class="na">abs</span><span class="p">(</span><span class="n">vector_Y</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">43</span><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">44</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">45</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">46</span><span class="cl"><span class="kt">int</span><span class="w"> </span><span class="n">final_distance</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">dist</span><span class="o">*</span><span class="n">distance</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">47</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">48</span><span class="cl"><span class="kd">final</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">final_X</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">final_distance</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln">49</span><span class="cl"><span class="kd">final</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="n">final_Y</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">vector_Y</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">150</span><span class="o">/</span><span class="n">255</span><span class="p">;</span></span></span></code></pre></div>
<p>The result was an input range of -150 and 150 for Y and between -255 and 255 for X, the values for Y were smaller because we didn’t want the car to move too fast but a full range of steering was crucial. Below are the results of a training session.</p>

<div><img src="training_data.webp" 
		 class="img-preset img-preset-md" width="600" height="400">
</div>

<p>Each dot on the above graph represent the location of the finger on the trackpad at the time that a training image was sliced from the video. The top part represents forward motion, the bottom backwards motion and when the car steered it produced dots that form the diagonals of this hourglass shaped graph. We cut the data into a few logical sectors,</p>
<ol>
<li>
<p>Forward</p>
</li>
<li>
<p>50% Forward</p>
</li>
<li>
<p>Forward Left</p>
</li>
<li>
<p>Forward Right</p>
</li>
<li>
<p>Backward</p>
</li>
<li>
<p>50% Backward</p>
</li>
<li>
<p>Backward Left</p>
</li>
<li>
<p>Backward Right</p>
</li>
</ol>
<p>And “Stop” which proved to be pretty disastrous because at some point the car would learn to permanently stop.</p>
<p>Collecting the image data was somewhat easier, all we had to do was record a video while the car was being driven. Here’s a link to a sample <a href="https://github.com/googlearchive/android-Camera2Video">android project</a> that records a video and saves it to storage when you’re done, we used the same code to capture our videos and even your ordinary camera app works if you don’t want to implement anything custom. The only meta data that’s required is the length of the video and when the recording was initiated.</p>
<h3 id="combining-inputs-and-labels">Combining Inputs and Labels</h3>
<p>In order to match the correct actions with what the car is actually seeing we have to bring the two data sets together. By splitting the training video captured by the front facing camera and time stamping it into separate images we can match them to the timestamp of a log from the controller’s input.</p>
<p>The video was split into 4 images per second of video, although with a 30fps recording we could have gone for a lot more detail. On the other hand our mobile device could capture hundreds of actions on the touchscreen per second, requiring us to disregard most of the data for picking one log per image from the video. Both data sets where submitted to a python script to slice the videos and create matching (image, log) pairs.</p>

<div><img src="footage.webp" 
		 class="img-preset img-preset-md" width="600" height="400">

	<figcaption style="margin:0px;" class="figure-caption">
		Each training frame receives one direction label
	</figcaption>
</div>

<p>After driving the car around our building for a bit, we gathered close to 7000 pairs of images and associated human control label pairs. Due our small budget and running the neural network on a phone, we had to down sample our images to 50x50 pixels so that our setup had a fighting chance of working in real time.</p>
<h3 id="using-neural-networks">Using Neural Networks</h3>
<p>This was arguably the most exciting and frustrating part of the project, the machine learning. Because we wanted the car to drive on its own we wanted the on board phone to accomplish both the seeing and the steering for the car. Both were indeed possible. We fully utilized the OpenCV Android library to obtain an image for every time the camera screen was updated as well as running a separate thread to control the Bluetooth chip on the car. We used this <a href="https://medium.com/android-news/a-beginners-guide-to-setting-up-opencv-android-library-on-android-studio-19794e220f3c">android project</a> as an introduction to OpenCV and it got us to a point were we could obtain individual frames of the camera in real time. All that was left to do was to complete a forward pass on our network model with images supplied by the library’s onCameraFrame() function every time there was a new frame.</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="kd">public</span><span class="w"> </span><span class="n">Mat</span><span class="w"> </span><span class="nf">onCameraFrame</span><span class="p">(</span><span class="n">CvCameraViewFrame</span><span class="w"> </span><span class="n">inputFrame</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w">  </span><span class="n">mGray</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">inputFrame</span><span class="p">.</span><span class="na">gray</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">  </span><span class="n">Mat</span><span class="w"> </span><span class="n">mEqualized</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">Mat</span><span class="p">(</span><span class="n">mGray</span><span class="p">.</span><span class="na">rows</span><span class="p">(),</span><span class="w"> </span><span class="n">mGray</span><span class="p">.</span><span class="na">cols</span><span class="p">(),</span><span class="w"> </span><span class="n">mGray</span><span class="p">.</span><span class="na">type</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w">  
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w">  </span><span class="c1">//normalize the image</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">  </span><span class="n">Imgproc</span><span class="p">.</span><span class="na">equalizeHist</span><span class="p">(</span><span class="n">mGray</span><span class="p">,</span><span class="w"> </span><span class="n">mEqualized</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w">  
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">  </span><span class="c1">//convert the frame into a scaled bitmap</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">  </span><span class="n">Bitmap</span><span class="w"> </span><span class="n">bitmap</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Bitmap</span><span class="p">.</span><span class="na">createBitmap</span><span class="p">(</span><span class="n">mEqualized</span><span class="p">.</span><span class="na">cols</span><span class="p">(),</span><span class="w"> </span><span class="n">mEqualized</span><span class="p">.</span><span class="na">rows</span><span class="p">(),</span><span class="w"> </span><span class="n">Bitmap</span><span class="p">.</span><span class="na">Config</span><span class="p">.</span><span class="na">RGB_565</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w">  </span><span class="n">Utils</span><span class="p">.</span><span class="na">matToBitmap</span><span class="p">(</span><span class="n">mEqualized</span><span class="p">,</span><span class="w"> </span><span class="n">bitmap</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w">  </span><span class="kd">final</span><span class="w"> </span><span class="n">Bitmap</span><span class="w"> </span><span class="n">scaled_bitmap</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Bitmap</span><span class="p">.</span><span class="na">createScaledBitmap</span><span class="p">(</span><span class="n">bitmap</span><span class="p">,</span><span class="w"> </span><span class="n">50</span><span class="p">,</span><span class="w"> </span><span class="n">50</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w">  </span><span class="c1">//get output from the neural net model</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="w">  </span><span class="kt">int</span><span class="o">[]</span><span class="w"> </span><span class="n">shape</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">model</span><span class="p">.</span><span class="na">get_Interpreter</span><span class="p">().</span><span class="na">getInputTensor</span><span class="p">(</span><span class="n">0</span><span class="p">).</span><span class="na">shape</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="w">  </span><span class="kd">final</span><span class="w"> </span><span class="n">ArrayList</span><span class="o">&lt;</span><span class="n">String</span><span class="o">&gt;</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">inferencer</span><span class="p">.</span><span class="na">multiImageInference</span><span class="p">(</span><span class="n">model</span><span class="p">,</span><span class="w"> </span><span class="n">scaled_bitmap</span><span class="p">,</span><span class="w"> </span><span class="n">shape</span><span class="o">[</span><span class="n">shape</span><span class="p">.</span><span class="na">length</span><span class="o">-</span><span class="n">1</span><span class="o">]</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="p">}</span></span></span></code></pre></div>
<p>At this point we could either write our own native neural network , or we could use Google’s <a href="https://ai.google.dev/edge/litert">Tensorflow Lite</a> library to load a python Keras model directly onto the phone. We went with Keras for the simplicity and existing collection of examples we could reference.</p>
<p>The only downside to this approach was that because the model file could no longer be changed on the Android phone, our car would not be able to use reinforcement learning to learn as it drove on its own. Fortunately this was a sacrifice we were willing to make.</p>
<p>With a basic Keras Sequential CNN model we pumped an input image of 2500 pixels into the series of Convolution and Max Pooling layers. We then converged the model into a dense layer at the end to give us 9 categories of output. For each output we would get a confidence value of how much the car was willing to follow any of the given directions. This was done because it’s not important for the car to make fine movements, the categories we specified above were enough.</p>
<h3 id="neural-network-architecture">Neural Network Architecture</h3>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Conv2D</span><span class="p">(</span><span class="mi">32</span><span class="p">,</span> <span class="n">kernel_size</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">                 <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">                 <span class="n">input_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">50</span><span class="p">,</span><span class="mi">50</span><span class="p">,</span><span class="mi">10</span><span class="p">),</span> 
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">                 <span class="n">name</span><span class="o">=</span><span class="s1">&#39;my_layer&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="n">convout1</span> <span class="o">=</span> <span class="n">Activation</span><span class="p">(</span><span class="s1">&#39;relu&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">convout1</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">MaxPooling2D</span><span class="p">(</span><span class="n">pool_size</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)))</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Conv2D</span><span class="p">(</span><span class="mi">64</span><span class="p">,</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">),</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="n">convout2</span> <span class="o">=</span> <span class="n">Activation</span><span class="p">(</span><span class="s1">&#39;relu&#39;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">convout2</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">MaxPooling2D</span><span class="p">(</span><span class="n">pool_size</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)))</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="mf">0.25</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Flatten</span><span class="p">())</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="mi">128</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Dropout</span><span class="p">(</span><span class="mf">0.5</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layers</span><span class="o">.</span><span class="n">Dense</span><span class="p">(</span><span class="n">numBins</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;softmax&#39;</span><span class="p">))</span></span></span></code></pre></div>
<p>Above is a Keras model we used for our car. It is relatively small and the gist of it is that it converts a 50x50 array of an input image into an array of 9 confidence outputs for each of the joystick colored sectors.</p>
<p>After a few weeks of tweaking we settled on this model that worked fairly well, giving us a 95% success rate with a sizable test set of 30% of our entire training database. The laststep was to save the Keras library as a tflite file and feed it to the Interpreter on the phone.</p>
<p>Tensorflow Lite has the amazing functionality of exporting the python written model into a tflite file which can be used by a Tensorflow Lite interpreter on any other system.</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="n">model_json</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">to_json</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">&#34;model.json&#34;</span><span class="p">,</span> <span class="s2">&#34;w&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">json_file</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="n">json_file</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">model_json</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="n">keras_file</span> <span class="o">=</span> <span class="s2">&#34;model.h5&#34;</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="n">model</span><span class="o">.</span><span class="n">save</span><span class="p">(</span><span class="n">keras_file</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="n">converter</span> <span class="o">=</span> <span class="n">lite</span><span class="o">.</span><span class="n">TFLiteConverter</span><span class="o">.</span><span class="n">from_keras_model_file</span><span class="p">(</span><span class="n">keras_file</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="n">tflite_model</span> <span class="o">=</span> <span class="n">converter</span><span class="o">.</span><span class="n">convert</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="nb">open</span><span class="p">(</span><span class="s2">&#34;model.tflite&#34;</span><span class="p">,</span> <span class="s2">&#34;wb&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">tflite_model</span><span class="p">)</span></span></span></code></pre></div>
<p>This model is simply loaded as a TFLite interpreter object which can then use the simple run() function on an input image to create a prediction.</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="kd">public</span><span class="w"> </span><span class="n">Interpreter</span><span class="w"> </span><span class="nf">load_internal_model</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">model_name</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">IOException</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w">        </span><span class="n">tfliteModel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">loadModelFile</span><span class="p">(</span><span class="n">model_name</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">        </span><span class="n">tflite_interpreter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">Interpreter</span><span class="p">(</span><span class="n">tfliteModel</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="n">tflite_interpreter</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="kd">private</span><span class="w"> </span><span class="n">ByteBuffer</span><span class="w"> </span><span class="nf">loadModelFile</span><span class="p">(</span><span class="n">String</span><span class="w"> </span><span class="n">model_name</span><span class="p">)</span><span class="w"> </span><span class="kd">throws</span><span class="w"> </span><span class="n">IOException</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">        </span><span class="n">AssetFileDescriptor</span><span class="w"> </span><span class="n">fileDescriptor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">activity</span><span class="p">.</span><span class="na">getAssets</span><span class="p">().</span><span class="na">openFd</span><span class="p">(</span><span class="n">model_name</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">        </span><span class="n">FileInputStream</span><span class="w"> </span><span class="n">inputStream</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="n">FileInputStream</span><span class="p">(</span><span class="n">fileDescriptor</span><span class="p">.</span><span class="na">getFileDescriptor</span><span class="p">());</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w">        </span><span class="n">FileChannel</span><span class="w"> </span><span class="n">fileChannel</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">inputStream</span><span class="p">.</span><span class="na">getChannel</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w">        </span><span class="kt">long</span><span class="w"> </span><span class="n">startOffset</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fileDescriptor</span><span class="p">.</span><span class="na">getStartOffset</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w">        </span><span class="kt">long</span><span class="w"> </span><span class="n">declaredLength</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">fileDescriptor</span><span class="p">.</span><span class="na">getDeclaredLength</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w">        </span><span class="k">return</span><span class="w"> </span><span class="n">fileChannel</span><span class="p">.</span><span class="na">map</span><span class="p">(</span><span class="n">FileChannel</span><span class="p">.</span><span class="na">MapMode</span><span class="p">.</span><span class="na">READ_ONLY</span><span class="p">,</span><span class="w"> </span><span class="n">startOffset</span><span class="p">,</span><span class="w"> </span><span class="n">declaredLength</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="p">}</span></span></span></code></pre></div>
<p>Now taking this model we can saved it into the memory of our phone and run it with the TensorFlow Interpreter to simply do a forward pass on the model.</p>






<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="kt">float</span><span class="o">[][][][]</span><span class="w"> </span><span class="n">float_pixels</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="kt">float</span><span class="o">[</span><span class="n">1</span><span class="o">][</span><span class="n">50</span><span class="o">][</span><span class="n">50</span><span class="o">][</span><span class="n">num_saved</span><span class="o">]</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="kt">float</span><span class="o">[][]</span><span class="w"> </span><span class="n">outputVal</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">new</span><span class="w"> </span><span class="kt">float</span><span class="o">[</span><span class="n">1</span><span class="o">][</span><span class="n">9</span><span class="o">]</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="k">for</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">0</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">50</span><span class="p">;</span><span class="w"> </span><span class="n">i</span><span class="o">++</span><span class="p">){</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w">    </span><span class="k">for</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">0</span><span class="p">;</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">50</span><span class="p">;</span><span class="w"> </span><span class="n">j</span><span class="o">++</span><span class="p">){</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">        </span><span class="k">for</span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">0</span><span class="p">;</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">num_saved</span><span class="p">;</span><span class="w"> </span><span class="n">b</span><span class="o">++</span><span class="p">){</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w">            </span><span class="n">float_pixels</span><span class="o">[</span><span class="n">0</span><span class="o">][</span><span class="n">i</span><span class="o">][</span><span class="n">j</span><span class="o">][</span><span class="n">b</span><span class="o">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="kt">float</span><span class="p">)</span><span class="w"> </span><span class="n">int_bitmaps</span><span class="p">.</span><span class="na">get</span><span class="p">(</span><span class="n">b</span><span class="p">)</span><span class="o">[</span><span class="n">i</span><span class="o">*</span><span class="n">50</span><span class="o">+</span><span class="n">j</span><span class="o">]/</span><span class="n">255</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">        </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="k">try</span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w">    </span><span class="n">tflite_interpreter</span><span class="p">.</span><span class="na">run</span><span class="p">(</span><span class="n">float_pixels</span><span class="p">,</span><span class="n">outputVal</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="p">}</span><span class="k">catch</span><span class="w"> </span><span class="p">(</span><span class="n">Exception</span><span class="w"> </span><span class="n">e</span><span class="p">){</span><span class="w">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="w">    </span><span class="n">e</span><span class="p">.</span><span class="na">printStackTrace</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="p">}</span><span class="w">
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="o">//</span><span class="n">outputVal</span><span class="w"> </span><span class="n">is</span><span class="w"> </span><span class="n">now</span><span class="w"> </span><span class="n">contains</span><span class="w"> </span><span class="n">confidence</span><span class="w"> </span><span class="n">values</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">all</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">sectors</span></span></span></code></pre></div>
<p>The sector with the highest confidence was taken as the action to execute by the car and finally all the pieces were in place.</p>
<div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;">
      <iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/F036dy1oTcA?autoplay=0&amp;controls=1&amp;end=0&amp;loop=0&amp;mute=0&amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"></iframe>
    </div>

<h3 id="conclusion">Conclusion</h3>
<p>At this point we had the car built, data collection set up and a neural network dictating the controls. The car was driving with the expertise of a 6 year old with a controller. It can safely be said that we were not winning any Grand Prix any time soon but the car was able to drive around on its own without crashing itself, with surprising reliability.</p>
<p>In any case we were happy that something that we’ve built over a semester while taking our other classes could actually learn on its own and accomplish such a difficult task, with minimal input from us. As a first project in machine learning this was a tough one to start with but looking back at it, the approach we took was in line with any other standard machine learning task.</p>
<p>If you’re still reading, hopefully you’ve learned something new and if not at least enjoyed our amateur struggles. Again, all the code is available here <a href="https://github.com/RoboticsCourse">https://github.com/RoboticsCourse</a>, please consider starring it and checking out the contributors.</p>
]]></content:encoded></item></channel></rss>