Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| reinforcedlearningwithaisupport:start [2025/03/08 16:58] – gedbadmin | reinforcedlearningwithaisupport:start [2025/03/08 17:14] (current) – gedbadmin | ||
|---|---|---|---|
| Line 21: | Line 21: | ||
| So I've decided to jump in and explore, and keep a journal along the way. This is because I only a few hours here and there to do this, so I want to be able to read back on what I did last time before moving forward. | So I've decided to jump in and explore, and keep a journal along the way. This is because I only a few hours here and there to do this, so I want to be able to read back on what I did last time before moving forward. | ||
| - | ===== Proof of Concept: Can I run SQLite | + | ===== A personal experiment - An SQLite |
| - | Last time I looked into this I tried taking the [[https:// | ||
| - | If I'm to embed SQLite within TiddlyWiki I need to run it locally as a standalone, so I ask ChatGPT if it can be done: | + | |
| - | + | ||
| - | >Is it possible to run SqlIte in the browser from a file, without any web server? | + | |
| - | + | ||
| - | >> | + | |
| - | >> | + | |
| - | >> | + | |
| - | >>* **Download SQLite Wasm** | + | |
| - | >>* **Create an HTML file** | + | |
| - | <code html> | + | |
| - | < | + | |
| - | <html lang=" | + | |
| - | < | + | |
| - | <meta charset=" | + | |
| - | <meta name=" | + | |
| - | < | + | |
| - | </ | + | |
| - | < | + | |
| - | < | + | |
| - | <input type=" | + | |
| - | <pre id=" | + | |
| - | + | ||
| - | <script src=" | + | |
| - | < | + | |
| - | let db; | + | |
| - | + | ||
| - | document.getElementById(' | + | |
| - | const file = event.target.files[0]; | + | |
| - | if (file) { | + | |
| - | const buffer = await file.arrayBuffer(); | + | |
| - | const SQL = await initSqlJs({ locateFile: filename => `https:// | + | |
| - | db = new SQL.Database(new Uint8Array(buffer)); | + | |
| - | + | ||
| - | const result = db.exec(" | + | |
| - | document.getElementById(' | + | |
| - | } | + | |
| - | }); | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | >>* **Open the file in a browser** | + | |
| - | >> | + | |
| - | >>// | + | |
| - | >>* The SQLite Wasm engine processes the .sqlite file. | + | |
| - | >>* Queries run directly in the browser. | + | |
| - | >>* The output shows available tables — you can expand this to support custom queries. | + | |
| - | + | ||
| - | I follow the instructions but I hit a problem. | + | |
| - | + | ||
| - | ===== I don't want a local database file ===== | + | |
| - | + | ||
| - | > Is a file necessary? | + | |
| - | >> Yes! You can create a purely in-memory SQLite database in the browser — no file needed. Here’s a simple example using sql.js: | + | |
| - | <code html> | + | |
| - | < | + | |
| - | <html lang=" | + | |
| - | < | + | |
| - | <meta charset=" | + | |
| - | <meta name=" | + | |
| - | < | + | |
| - | </ | + | |
| - | < | + | |
| - | < | + | |
| - | <button id=" | + | |
| - | <pre id=" | + | |
| - | + | ||
| - | <script src=" | + | |
| - | < | + | |
| - | (async () => { | + | |
| - | const SQL = await initSqlJs({ locateFile: file => `https:// | + | |
| - | const db = new SQL.Database(); | + | |
| - | + | ||
| - | document.getElementById(' | + | |
| - | db.run(" | + | |
| - | db.run(" | + | |
| - | + | ||
| - | const result = db.exec(" | + | |
| - | document.getElementById(' | + | |
| - | }); | + | |
| - | })(); | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | + | ||
| - | I run this and it works, giving me the following result set in a JSON format: | + | |
| - | <code javascript> | + | |
| - | [ | + | |
| - | { | + | |
| - | " | + | |
| - | " | + | |
| - | " | + | |
| - | ], | + | |
| - | " | + | |
| - | [ | + | |
| - | 1, | + | |
| - | " | + | |
| - | ], | + | |
| - | [ | + | |
| - | 2, | + | |
| - | " | + | |
| - | ] | + | |
| - | ] | + | |
| - | } | + | |
| - | ] | + | |
| - | </code> | + | |
| - | + | ||
| - | ===== Loading WASM from a Base64 encoded string ===== | + | |
| - | + | ||
| - | It's still loading code and WASM from a server. | + | |
| - | + | ||
| - | > I'd like to run this offline. | + | |
| - | >> Yes! You can absolutely run SQLite (with sql.js) offline by downloading the necessary files. Let’s set this up step by step. | + | |
| - | + | ||
| - | The instructions a straight forward. | + | |
| - | + | ||
| - | <code javascript> | + | |
| - | // Initialize sql.js with local wasm file | + | |
| - | initSqlJs({ | + | |
| - | locateFile: file => `lib/ | + | |
| - | }).then(SQL => { ... } | + | |
| - | </ | + | |
| - | + | ||
| - | This is when I run into the same CORS Policy problems that blocked me in the past: | + | |
| - | + | ||
| - | < | + | |
| - | Access to fetch at ' | + | |
| - | + | ||
| - | Failed to load resource: net:: | + | |
| - | + | ||
| - | Uncaught (in promise) TypeError: Failed to fetch (sql-wasm.js: | + | |
| - | at sql-wasm.js: | + | |
| - | at sql-wasm.js: | + | |
| - | at sql-wasm.js: | + | |
| - | at new Promise (< | + | |
| - | at initSqlJs (sql-wasm.js: | + | |
| - | at Test.html: | + | |
| - | at Test.html: | + | |
| - | </ | + | |
| - | + | ||
| - | I tell ChatGPT about the error, and it confidently provides me with a fix. | + | |
| - | + | ||
| - | ===== Fixing the CORS issue ===== | + | |
| - | + | ||
| - | >> Ah, yes — this happens because modern browsers block fetch requests for local files due to CORS restrictions. Even though you’re running things locally, loading '' | + | |
| - | >> Let’s fix it! 🚀 | + | |
| - | + | ||
| - | It then gives me three possible options: | + | |
| - | 1. Run a simple local server (Recommended) | + | |
| - | 1. Use a VS Code extension (even simpler) | + | |
| - | 1. Not recommended) Disable CORS checks temporarily | + | |
| - | + | ||
| - | None of these really work for me. For the WASM to work in TiddlyWiki I'll need to load the content from within the file. This is how TiddlyWiki manages all of it's resources, including Javascript. | + | |
| - | + | ||
| - | I explain my problem to ChatGPT, and it confidently provides the answer: | + | |
| - | + | ||
| - | > I don't really like any of these options. | + | |
| - | > Is there anyway of loading WASM from data embedded in my HTML file, perhaps BASE64 encoded? | + | |
| - | >> Yes! You can absolutely embed the WebAssembly (WASM) directly into your TiddlyWiki without needing any external files or servers. Let’s tackle this step by step. 🚀 | + | |
| - | + | ||
| - | I am loving this AI's can do attitude, so I follow the steps provided. | + | |
| - | + | ||
| - | First of all I convert my WASM file to Base64. | + | |
| - | <code powershell> | + | |
| - | [convert]:: | + | |
| - | </ | + | |
| - | + | ||
| - | Then I copy the Base64 text into a string so that I can loading it into a binary array. | + | |
| - | + | ||
| - | One thing I like at this point is that ChatGPT doesn' | + | |
| - | + | ||
| - | <code javascript> | + | |
| - | < | + | |
| - | <html lang=" | + | |
| - | < | + | |
| - | <meta charset=" | + | |
| - | < | + | |
| - | </ | + | |
| - | < | + | |
| - | < | + | |
| - | <script src=" | + | |
| - | < | + | |
| - | // Paste the long Base64-encoded WASM here | + | |
| - | const wasmBase64 = " | + | |
| - | + | ||
| - | // Convert Base64 to binary | + | |
| - | const binaryString = atob(wasmBase64); | + | |
| - | const bytes = new Uint8Array(binaryString.length); | + | |
| - | for (let i = 0; i < binaryString.length; | + | |
| - | bytes[i] = binaryString.charCodeAt(i); | + | |
| - | } | + | |
| - | const wasmBinary = bytes.buffer; | + | |
| - | + | ||
| - | // Initialize sql.js using the in-memory WASM binary | + | |
| - | initSqlJs({ | + | |
| - | locateFile: () => URL.createObjectURL(new Blob([wasmBinary], | + | |
| - | }).then(SQL => { | + | |
| - | const db = new SQL.Database(); | + | |
| - | db.run(" | + | |
| - | db.run(" | + | |
| - | + | ||
| - | const res = db.exec(" | + | |
| - | console.log(" | + | |
| - | }).catch(err => { | + | |
| - | console.error(" | + | |
| - | }); | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | + | ||
| - | There are two problems with this code. | + | |
| - | + | ||
| - | - ChatGPT has decided to drop functionality from the code we are writing. | + | |
| - | - It doesn' | + | |
| - | + | ||
| - | > I'm getting the error: wasm streaming compile failed: TypeError: Failed to execute ' | + | |
| - | + | ||
| - | And now we see how ChatGPT shines as a research assistant and guide, because it immediately identified the problem and provides the fix: | + | |
| - | + | ||
| - | ===== Fixing the MIME type ===== | + | |
| - | + | ||
| - | >>Ah, that's happening because the WebAssembly streaming compiler expects the MIME type to be application/ | + | |
| - | >> | + | |
| - | >> | + | |
| - | + | ||
| - | It gives me a full page of code, but the only part that I'm interested in is this: | + | |
| - | < | + | |
| - | // Initialize sql.js using the in-memory WASM binary | + | |
| - | initSqlJs({ | + | |
| - | locateFile: () => { | + | |
| - | // Create a blob for the binary and generate a URL | + | |
| - | const blob = new Blob([bytes], { type: " | + | |
| - | return URL.createObjectURL(blob); | + | |
| - | } | + | |
| - | }).then(SQL => { ... }); | + | |
| - | </ | + | |
| - | + | ||
| - | I list this %%locateFile%% definition, so that my final code looks like this: | + | |
| - | + | ||
| - | <code html> | + | |
| - | < | + | |
| - | <html lang=" | + | |
| - | < | + | |
| - | <meta charset=" | + | |
| - | <meta name=" | + | |
| - | < | + | |
| - | </ | + | |
| - | < | + | |
| - | < | + | |
| - | <button id=" | + | |
| - | <pre id=" | + | |
| - | + | ||
| - | <script src=" | + | |
| - | < | + | |
| - | const base64wasm = " | + | |
| - | const binaryString = atob(base64wasm); | + | |
| - | const bytes = new Uint8Array(binaryString.length); | + | |
| - | for (let i = 0; i < binaryString.length; | + | |
| - | bytes[i] = binaryString.charCodeAt(i); | + | |
| - | } | + | |
| - | + | ||
| - | (async () => { | + | |
| - | // Initialize sql.js using the in-memory WASM binary | + | |
| - | const sql = await initSqlJs({ | + | |
| - | locateFile: () => { | + | |
| - | // Create a blob for the binary and generate a URL | + | |
| - | const blob = new Blob([bytes], | + | |
| - | return URL.createObjectURL(blob); | + | |
| - | } | + | |
| - | }); | + | |
| - | + | ||
| - | const db = new sql.Database(); | + | |
| - | + | ||
| - | document.getElementById(' | + | |
| - | + | ||
| - | db.run(" | + | |
| - | db.run(" | + | |
| - | + | ||
| - | const result = db.exec(" | + | |
| - | document.getElementById(' | + | |
| - | }); | + | |
| - | })(); | + | |
| - | + | ||
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | </ | + | |
| - | + | ||
| - | This completes my proof of concept. | + | |
| - | + | ||
| - | The code is simple, and I mostly understand what it is doing. | + | |
| - | + | ||
| - | >> **✅ Why this works:** | + | |
| - | >> | + | |
| - | >> | + | |
| - | >> | + | |
| - | + | ||
| - | That isn't enough for me. I want to know more. | + | |
| - | + | ||
| - | ===== Understanding URL.createObjectURL() ===== | + | |
| - | + | ||
| - | > Could you tell me more about what URL.createObjectURL does? | + | |
| - | + | ||
| - | >> Absolutely! Let’s break down what URL.createObjectURL() does and why it’s useful, especially for what we’re doing. | + | |
| - | >> | + | |
| - | >> **🔥 What is URL.createObjectURL? | + | |
| - | >> %%URL.createObjectURL()%% is a method that takes a Blob or File object and returns a temporary URL that points to the data in memory. ??>> This allows you to treat in-memory data — like a WASM binary — as if it were a file on a server. | + | |
| - | >> | + | |
| - | >> **Syntax: | + | |
| - | >> <code javascript> | + | |
| - | const blob = new Blob([data], | + | |
| - | const url = URL.createObjectURL(blob); | + | |
| - | </ | + | |
| - | >> | + | |
| - | >> The %%url%% now looks something like this: | + | |
| - | >> < | + | |
| - | blob: | + | |
| - | </ | + | |
| - | >> | + | |
| - | >> **✅ Why use it here?** | + | |
| - | >> When loading WebAssembly, | + | |
| - | >> | + | |
| - | >> By converting the binary into a Blob, and then using createObjectURL, | + | |
| - | >> | + | |
| - | >> | + | |
| - | >> | + | |
| - | >> | + | |
| - | >> | + | |
| - | >> ** Cleaning up the URL:** | + | |
| - | >> These blob URLs are only valid while the page is open. If you don’t need them anymore (to free up memory): | + | |
| - | >> | + | |
| - | >> <code javascript> | + | |
| - | URL.revokeObjectURL(url); | + | |
| - | </ | + | |
| - | >> | + | |
| - | >> You’d typically do this after the WebAssembly module is loaded. | + | |
| - | + | ||
| - | That's a good explanation. | + | |
| - | + | ||
| - | My next question is about that final point, about clean up. | + | |
| - | + | ||
| - | ===== Clearning up the object URL ===== | + | |
| - | + | ||
| - | > Looking back at the SQLite code, we don’t cleanup our blob url. | + | |
| - | + | ||
| - | >> You're absolutely right — we didn't clean up the blob URL after loading the SQLite WASM module. Let’s fix that! | + | |
| - | >> | + | |
| - | >> In the original code, we did something like this: | + | |
| - | >> | + | |
| - | >>< | + | |
| - | const blob = new Blob([wasmBinary], | + | |
| - | const url = URL.createObjectURL(blob); | + | |
| - | + | ||
| - | const sqlPromise = initSqlJs({ locateFile: () => url }); | + | |
| - | + | ||
| - | sqlPromise.then(SQL => { | + | |
| - | const db = new SQL.Database(); | + | |
| - | console.log(' | + | |
| - | }); | + | |
| - | </ | + | |
| - | >> | + | |
| - | >> **✅ Adding cleanup: | + | |
| - | >> We should revoke the blob URL once the WASM module is loaded: | + | |
| - | >> | + | |
| - | >> <code javascript> | + | |
| - | const blob = new Blob([wasmBinary], | + | |
| - | const url = URL.createObjectURL(blob); | + | |
| - | + | ||
| - | const sqlPromise = initSqlJs({ | + | |
| - | locateFile: () => url | + | |
| - | }); | + | |
| - | + | ||
| - | sqlPromise.then(SQL => { | + | |
| - | const db = new SQL.Database(); | + | |
| - | console.log(' | + | |
| - | URL.revokeObjectURL(url); | + | |
| - | }); | + | |
| - | </ | + | |
| - | >> **✅ Why clean up?** | + | |
| - | >> Memory leaks: Blobs can stick around in memory until you explicitly revoke them or the page is closed. | + | |
| - | >> Best practice: Keeping your resources tidy is just good hygiene — especially for long-running apps like TiddlyWiki. | + | |
| - | >> Would you like me to slip this fix into the plugin code? Let me know if you want the cleanup tied to a particular lifecycle event! 🚀 | + | |
| - | + | ||
| - | This is good guidance, and the implementation approach is well described, so I add the clean up code. It creates a promise, which we already have in our code so I ask: | + | |
| - | + | ||
| - | > initSwlJs() appears to return a promise. | + | |
| - | + | ||
| - | >> Yes, exactly. Since initSqlJs() returns a promise, you can add the cleanup in the .then() callback once the module is loaded. Here’s an updated code snippet that demonstrates this: | + | |
| - | >> | + | |
| - | >> <code javascript> | + | |
| - | // Convert the Base64-encoded WASM string to a binary array | + | |
| - | const wasmBase64 = ` | + | |
| - | BASE64_LINE_1 | + | |
| - | BASE64_LINE_2 | + | |
| - | BASE64_LINE_3 | + | |
| - | `.replace(/ | + | |
| - | const binaryString = atob(wasmBase64); | + | |
| - | const bytes = new Uint8Array(binaryString.length); | + | |
| - | for (let i = 0; i < binaryString.length; | + | |
| - | bytes[i] = binaryString.charCodeAt(i); | + | |
| - | } | + | |
| - | const wasmBinary = bytes.buffer; | + | |
| - | + | ||
| - | // Create a Blob from the binary data and get a blob URL | + | |
| - | const blob = new Blob([wasmBinary], | + | |
| - | const blobUrl = URL.createObjectURL(blob); | + | |
| - | + | ||
| - | // Initialize sql.js using the in-memory WASM binary | + | |
| - | initSqlJs({ | + | |
| - | locateFile: () => blobUrl | + | |
| - | }).then(SQL => { | + | |
| - | // The SQL module is ready to use. | + | |
| - | const db = new SQL.Database(); | + | |
| - | db.run(" | + | |
| - | db.run(" | + | |
| - | + | ||
| - | const res = db.exec(" | + | |
| - | console.log(" | + | |
| - | + | ||
| - | // Cleanup the blob URL after loading the WASM module | + | |
| - | URL.revokeObjectURL(blobUrl); | + | |
| - | }).catch(err => { | + | |
| - | console.error(" | + | |
| - | }); | + | |
| - | </ | + | |
| - | >> | + | |
| - | >> **Explanation** | + | |
| - | >> **Blob and URL:** We create a Blob from the WASM binary with the correct MIME type (%%application/ | + | |
| - | >> **initSqlJs**: | + | |
| - | >> **Cleanup**: | + | |
| - | >> | + | |
| - | >> This approach keeps everything tidy and ensures that the blob URL is only kept for as long as needed. Let me know if you have any more questions! | + | |
| - | + | ||
| - | Eager to please the chatbot has followed my suggestion and used the %%then%% block. | + | |
| - | + | ||
| - | > Will cleanup occur if an exception is thrown? | + | |
| - | + | ||
| - | >> By default, if you only put cleanup in the %%.then()%% block, an exception in the promise chain might bypass it. To ensure that the blob URL is always revoked regardless of success or failure, you can use the %%.finally()%% method, which runs whether the promise resolves or rejects. Here’s an updated example: | + | |
| - | >> | + | |
| - | >> <code javascript> | + | |
| - | initSqlJs({ | + | |
| - | locateFile: () => blobUrl | + | |
| - | }).then(SQL => { | + | |
| - | const db = new SQL.Database(); | + | |
| - | db.run(" | + | |
| - | db.run(" | + | |
| - | const res = db.exec(" | + | |
| - | console.log(" | + | |
| - | }).catch(err => { | + | |
| - | console.error(" | + | |
| - | }).finally(() => { | + | |
| - | // This will run whether the promise was fulfilled or rejected. | + | |
| - | URL.revokeObjectURL(blobUrl); | + | |
| - | }); | + | |
| - | </ | + | |
| - | >> | + | |
| - | >> In this setup, if an exception is thrown at any point during initialization, | + | |
| - | + | ||
| - | I'm happy with this code. After a couple of hours at the keyboard I call it a night. | + | |
| - | + | ||
| - | ===== Tidying Up ===== | + | |
| - | + | ||
| - | I just want two more things to complete my proof of concept. | + | |
| - | - A test area where I can input the SQL to be executed. | + | |
| - | - Display the response as a table instead of JSON. | + | |
| - | + | ||
| - | Since I am ignoring most of the code being generated by ChatGPT, I ask it to be more concise. | + | |
| - | + | ||
| - | > From now on, just show me the enough code snippet for me to alter the code myself. | + | |
| - | + | ||
| - | It obliges, and repeats the saliant code for the cleanup. | + | |
| - | + | ||
| - | The first item is my text area. | + | |
| - | + | ||
| - | > I currently have a standalone html file that implements this. | + | |
| - | > | + | |
| - | > Could you give me the code to show a TEXTAREA where the query is entered, and the code to extract the sql to be run? | + | |
| - | + | ||
| - | >> <code javascript> | + | |
| - | < | + | |
| - | <button id=" | + | |
| - | < | + | |
| - | document.getElementById(" | + | |
| - | const sqlQuery = document.getElementById(" | + | |
| - | // Now use sqlQuery, e.g.: | + | |
| - | // db.run(sqlQuery); | + | |
| - | console.log(" | + | |
| - | }); | + | |
| - | </ | + | |
| - | </ | + | |
| - | + | ||
| - | I copy this code into my page, adapt it slightly and it works. | + | |
| - | + | ||
| - | > Finally, could you give me some code to take the Sqlite result and turn it into a html table? | + | |
| - | + | ||
| - | >> <code javascript> | + | |
| - | function resultToHTMLTable(result) { | + | |
| - | if (!result || result.length === 0) { | + | |
| - | return "< | + | |
| - | } | + | |
| - | // Use the first result set (if there are multiple) | + | |
| - | const data = result[0]; | + | |
| - | let html = "< | + | |
| - | + | ||
| - | // Create header row | + | |
| - | data.columns.forEach(col => { | + | |
| - | html += `< | + | |
| - | }); | + | |
| - | html += "</ | + | |
| - | + | ||
| - | // Create data rows | + | |
| - | data.values.forEach(row => { | + | |
| - | html += "< | + | |
| - | row.forEach(cell => { | + | |
| - | html += `< | + | |
| - | }); | + | |
| - | html += "</ | + | |
| - | }); | + | |
| - | + | ||
| - | html += "</ | + | |
| - | return html; | + | |
| - | } | + | |
| - | + | ||
| - | // Example usage: | + | |
| - | // Assuming ' | + | |
| - | // let result = db.exec(" | + | |
| - | // Then insert the table into an element: | + | |
| - | // document.getElementById(" | + | |
| - | </ | + | |
| - | + | ||
| - | Again, this is easily copied and adapted into my code. | + | |
| - | + | ||
| - | ===== Conclusion ===== | + | |
| - | + | ||
| - | I've heard a lot about how LLMs about the productivity gains from using AI as a co-pilot. | + | |
| - | + | ||
| - | The approach I have had great success with is the one used here. Leaning on AI to help me understand the libraries I am working with. After establishing the initial file, I think ask the Chatbot for the information I need to implement the next feature. | + | |
| - | + | ||
| - | Rather than asking the LLM to write code for me, I use it to reinforce my learning. | + | |
| - | + | ||
| - | This is how AI will boost developer productivity: | + | |