This short but picture-filled post is intended to explain the process of building this website - from the languages it's written in to the platforms it's hosted on. While it will be fairly high-level, I'll link to some great resources for building your own should that pique your interest.

1. Why build a website?

I originally wanted to build this to host always-on scripts that could run periodically without my computer being on. I also happened to be experimenting with embedding web applications within a web page - rather than as a separate site. You can see some examples of them here and here. I'm a big fan of allowing people to play around with data and models, rather than just reading about them, because it's engaging and you learn more.

I then thought hey, why not take the opportunity to write up some of the side projects I work on into proper posts. This had the twin advantage of both having a portfolio of sorts to look back on my work, and forcing me to finish them all properly before moving onto the next one. I've only written up a selection of them, but over time this will grow.

I'm not unaware of the irony of starting a new project in order to force myself to write up old projects, and to an extent of course it was also just a fun excuse to learn something new. As you'll see here it involved a real breadth of learning (from css to cloud databases) that was a lot of fun, and I'm glad I did!

2. The current structure

They say a picture tells a thousand words, so I made this diagram and will endeavour not to write out every element of its contents. Suffice to say there are quite a few different aspects of this website, many of which I had to learn from scratch, but they fall into two main categories: building it and hosting it. The short answer is - the back end is written in Python and hosted on AWS, but with some complexities like a database and interactive web apps that I'll explore below as I take you through how it developed.

3. The initial site

The back end of the website is written in Python, using a framework called Flask - it's the most popular Python web framework and is simpler and more lightweight than its main competitor Django. I chose Python because I know Python, it's fun, and because I intended on using its strength for machine learning when building more interactive apps in Python it was a natural fit.

The very first iteration of the site was incredibly simple - I used this excellent guide to get going and built the most basic first flask site possible to get it up and running, with only two pages and basically no formatting:

It was hosted on my computer, so I could run it locally but noone else could access it.

4. Building the front end

The next step was to make it look a little less like it was made in the 1990s. This involves a lot of HTML, CSS and Javascript - which I'd never used before. As a result I relied fairly heavily on existing Bootstrap templates which I modified rather than writing from scratch.

You can see here the first layout I tried (complete with my first attempt at a javascript graph linked to a fantasy football API), before deciding it was more suited for an online resume than a fully fledged website with apps and a blog:

I eventually settled on this bootstrap template initio which you can see here:

You'll probably notice quite a few similarities to how the website currently looks - from the menu bar to the profile picture in a circle. While I've made quite a few adjustments as I've grown more comfortable playing around with the HTML and CSS, I've retained a lot of the functionality of this layout.

5. Adding log in options and a local database blog

The database part is where it gets interesting - I quickly realised that if I needed a separate html page for every single post it was going to get messy fast. This would have a load of disadvantages - not least that I could only edit things from my computer and I'd have to literally write them in an HTML document which is messy. For ten posts I'd have ten separate files - not ideal. So I looked into setting up a database to hold them.

Effectively what I set up was a database that holds the title and text of blog posts, and then an HTML page that basically slots in the text of whichever blog post the user has chosen. Initially this database was local (using SqLite) but as you'll see I ended up hosting it on AWS.

Of course that brings its own challenges and opportunities - if you're running it all off a database, you can let blog posts be made from the website itself. But that requires adding log in functionality. So I set that up too, and you can see below the view I have when logged in:

This means I can write and edit posts, save drafts, have a testing area - all without compromising the experience for normal visitors. I can critically also write and post from anywhere.

I'd recommend checking out this great guide for using a database to hold blog entries that I used extensively to help with this.

6. Hosting the website on AWS

Having the website available all the time was half the point of making it in the first place, so I turned to AWS' Elastic Beanstalk product to get it online. This involved only a small amount of rework, and I started off with the simple approach of uploading a zipped folder containing the website.

The problem with going to the AWS website every time to upload the file is that it takes time (and forces you to make a bunch of zip files). So after getting mildly annoyed by the five minutes it took to do every time I wanted to deploy my changes, I switched over to using git and the command line. For those unfamiliar with git, it's basically a version control that allows me to push any changes I make to an online copy of my code. It will check what's changed, I can have different versions, I can see my change history and it's all very convenient. I can also do it from the command line, which you can see in this image. We have:

1. A command to temporarily load up all my changes: git add .

2. A command to commit all those loaded up changes to the local copy of my code: git commit -m "with a message"

3. A command to push my local code to the online version on AWS Codecommit: git push codecommit-origin

And with three commands all my changes are now on the online version, and all that's left to do is 'deploy' that online copy of the code and make it the new code for the site itself on Elastic Beanstalk with the two word command eb deploy:

So now, with four lines of code I not only have a fully tracked history of what I've changed, but I've also updated the actual website on the AWS servers - all without having to create any zip files or open any browsers!

7. Hosting the database on AWS

Initially the database for the blog posts was an actual file in a zip folder, but as I wanted to be able to interact with it separately I decided to host it separately on AWS using their Relational Database Service (RDS):

This meant I could access it remotely with Microsoft Server Management Studio and a bit of SQL which was far cleaner:

8. Getting the website domain with AWS

Now my website works, it's hosted online, it's easy to update - but its web address is http://crkershaw.us-east-2.elasticbeanstalk.com/. This is the actual web address of this website - not very catchy. To fix this is very easy, and goes back to the fundamentals of how the internet works - the Domain Name System (DNS).

DNS is basically a decentralised system that converts a website address (e.g. charliekershaw.com) into its underlying IP address (e.g. 18.188.63.189). In this case, I bought the charliekershaw.com web address from amazon using their Route 53 service, and set it up to point to the same IP address as the crkershaw.us-east-2.elasticbeanstalk.com URL. You can actually test this out yourself by pinging them:

1. Open the command line (hold down windows key + r, type cmd and hit enter)

2. Type ping www.charliekershaw.com and hit enter. You should see it automatically pulls the IP address 18.188.63.189

3. Then do the same with ping crkershaw.us-east-2.elasticbeanstalk.com - you should get the same IP address

If you don't want to try it yourself, this is what it looks like:

9. Embedded applications

The final step is adding content. Some of this comes in the form of writing the blog posts, but another big part is deploying interactive applications so visitors can play around with the output. Shiny is a web app framework for R that allows you to spin up interactive, web-based applications. I've built a lot of Shiny Apps in my time, including some very complex ones that act as a full standalone application. Most Shiny Apps don't need that level of complexity though, and are suitable for embedding in a web page. The big advantage of that is the user doesn't have to leave the site or even the page to play around with it.

The mini apps I'm publicly hosting are currently hosted separately on shinyapps.io and just embedded in the page using something called an iframe. This isn't ideal for speed, but it's very easy to spin up and down. You can check them out in the Apps dropdown in the menu bar!

I'm hoping to build out future apps in Dash (basically Shiny for Python) and straight up Javascript embedded in the page - but it's another thing on my very long list of projects to do.

10. Future developments

So that's the website! From locally hosted 90s look and feel, to a fully fleshed out front and back end with everything from authentication to database and web-app integration. As this acts as a bit of a testing platform for me, I hope I can keep developing it and its content so it will never be fully completed, but I've learned a huge amount along the way and have a lot of areas I want to explore - both in content to add and architectural improvements!