DataScienceCastnet

An archive of my blog posts, originally posted on datasciencecastnet.home.blog.This may not be their permanent home.

Computational Linguistics

LLMs are cool - we can make them do all sorts of tricks. But beyond the generative applications there’s a whole bunch of interesting linguistic analysis we can do…

More=Better?

In this post I want to share a few quick experiments to show something that is both obvious and also perhaps underappreciated: you can get a sizeable quality boost from a…

A few tips for working on high-surface-area problems

What do you do when there are too many pieces to fit in your head?

Some of my posts will now be duplicated to Answer.ai’s blog since I work there now. This one has just gone up (12 April) but since it has sat as a draft since January I’m…

Quick Experiment: Filtering Preference Data for Better DPO

With recent discussion around DPO (and proposed alternatives cDPO and IPO) I figured it was time we ran some experiments to see what works and what doesn’t. As a good…

Exploring Softmax1, or “Community Research For The Win!”

Last week a guy called Evan Miller tweeted out a blog post claiming to have discovered a flaw in the attention mechanism used by transformers today:

Why and how I’m shifting focus to LLMs

While I’ve previously consulted on NLP projects, in the past few years my research focus has been chiefly on images. If you had asked me a few months ago about looking at…

A Recipe for Training Good Generative Models

Generative models are all the rage at the moment, and quality seems to be skyrocketing across the board. In this post, I share what I’m realizing is *the* key recipe that is…

DistilHN: Summarizing News Articles with Transformers

https://youtu.be/TBqzMfWpQrs

How Predictable: Evaluating Song Lyrics with Language Models

I was briefly nerd-sniped this morning by the following tweet:

Update Time

A few recent projects I’ve worked on have been documented elsewhere but haven’t made it to this blog. The point of this post is to summarize these so that they aren’t lost…

Fine-tuning a CLOOB-Conditioned Latent Diffusion Model on WikiArt

Prompt: ‘A sunset landscape painting, oil on canvas’ (fine-tuned Wikiart model)

Sketchy Unet

The model demo running on Huggingface Spaces

Turtle Recall: A Contrastive Learning Approach

NB: A scoring glitch caused this approach to look very good on the leaderboard, but local validation and a fix from Zindi later confirmed that it isn’t as magical as it…

AIAIART Course Retrospective

A small selection of images generated during the course

Playing with Tweet Sentiment Analysis

The average sentiment of the most recent 200 tweets from each country’s capital city.

WhistleGen: Generating Traditional Irish music with ML

https://www.youtube.com/watch?v=B7SEP2p4Y1A

In Brief: Playing with Class Imbalance

We often encounter imbalanced data in the world of machine learning, and have to decide how best to handle this. In ‘real life’ it is up to us to decide how to evaluate…

BirdClef Entry: Bird Call Classification with FastAI

The Cornell Lab of Ornithology run an annual competition to identify bird calls in soundscapes. I decided to have a go at this year’s competition to get back into audio…

Language Models for Protein Sequence Classification

A 3D model of a protein kinase

Data Glimpse: Predicted Historical Air Quality for African Cities

Air quality has been in the news a lot recently. Smoke from fires has had thousands of Californians searching for info around the health hazards of particulate matter…

Personal Metrics

This is just a quick post following on from some recent conversations in this area. tldr: Tracking some data about yourself is a great exercise, and I highly recommend it.…

Update: Why the pause?

It’s been nearly four months since my last post on this blog. I’d like to get back into writing things up here, but before putting up another normal post I figured it was…

Self-Supervised Learning with Image网

Until fairly recently, deep learning models needed a LOT of data to get decent performance. Then came an innovation called transfer learning, which we’ve covered in some…

Meta ‘Data Glimpse’ - Google Dataset Search

Christmas came in January this year, with Google’s release of ‘Dataset Search’. They’ve indexed millions of cool datasets and made it easy to search through them. This post…

Swoggle Part 2 - Building a Policy Network with PyTorch, dealing with Cheaty Agents and ‘Beating’ the Game

In part 1, we laid the groundwork for our Reinforcement Learning experiments by creating a simple game (Swoggle) that we’d be trying to teach out AI to play. We also created…

Swoggle Part 1- RL Environments and Literate Programming with NBDev

I’m going to be exploring the world of Reinforcement Learning. But there will be no actual RL in this post - that’s for part two. This post will do two things: describe the…

Behind the scenes of a Zindi Contest

Snapshot Serengeti - Working with Large Image Datasets

Driven Data launched a competition around the Snapshot Serengeti database - something I’ve been intending to investigate for a while. Although the competition is called…

Deep Learning + Remote Sensing - Using NNs to turn imagery into meaningful features

Every now and again, the World Bank conducts something called a Living Standards Measurement Study (LSMS) survey in different countries, with the purpose being to learn…

Zindi UberCT Part 3: Uber Movement

Uber Movement has launched in Cape Town

Zindi UberCT Part 2: Stepping Up

In part 1, we looked at the SANRAL challenge on Zindi and got a simple first submission up on the leaderboard. In this tutorial I’ll show some extra features you can add on…

Zindi UberCT Part 1: Getting started

Welcome to the first in a three-part series on Zindi’s Uber Movement SANRAL Cape Town Challenge. This tutorial will take a look at the challenge, start exploring the data…

Packaging a classification model as a web app

My shiny new web app, available here

Pothole Detection (aka Johno tries fastai)

This week saw folks from all over the AI space converge in Cape Town for the AI Expo. The conference was inspiring, and I had a great time chatting to all sorts of…

Trying Automated ML

Some students had asked me for my opinion on automated tools for machine learning. The thought occurred that I hadn’t done much with them recently, and it was about time I…

Mapping Change in Cropland in Zimbabwe (Part 2)

In Part 1, I showed a quick way to get a model that predicts cropland extent, using someone else’s model as a starting point. This was a fun exercise, but in today’s post…

Mapping Change in Cropland in Zimbabwe (Part 1)

I’m going to be working on a project that will ultimately require manually outlining tobacco fields in Zimbabwe. To help locate potential fields, it would be nice to have a…

Data Glimpse: Cropland and Settlement maps from QED.AI

The point of this Data Glimpse post is to feature a wonderful yet badly publicized data source: https://maps.qed.ai/. Using crowd-sourced data, they built really accurate…

Data Glimpse: Nighttime Lights

This ‘Data Glimpse’ post will look at the Global Radiance-Calibrated Nighttime Lights dataset [1], available through Google Earth Engine. However, the method shown here can…

Data Glimpse: South Africa’s Hydrological Data

South Africa’s Department of Water Affairs (DWA) makes all kinds of data publicly available through their data portal: http://www.dwa.gov.za/hydrology/. The download…

Data Glimpse: Visualizing Economic Activity with the G-Econ Project data

This is the first ‘data glimpse’ - a short exploration of an existing dataset, with code and examples showing some of the ways the data can be used. For today’s glimpse…

Tutorial: Improving Crop Type Predictions

Following on from the last tutorial, this post will look at some ways we can improve our crop identification method. At the end of the last post, we were using a CART…

Tutorial: Predicting Crop Types with GEE

Zindi is currently hosting a competition to classify fields by crop type using Sentinel-2 satellite imagery. They provide labeled fields with the crop type, and a separate…

New Database: Forest Change in Different Regions

Forest loss is a major problem facing many parts of the world right now. Trees are being cleared to make way for agriculture, or simply cut down for fuel and timber.…

Zindi Competition 1 - Making Art!

I’m going to try entering some Zindi competitions this week. First up is the ‘AI Art’ contest. I have many crazy plans, but my nascent tensorflow skills mean everything…

Zindi Competition 2 - Trying CatBoost on the Traffic Jam Challenge

Zindi ran a challenge predicting bus ticket sales into Nairobi. It is now closed, but we can still make predictions and see how they would have done. This was a very quick…

Looking at traffic/congestion vs air quality AKA a quest for data

I’m currently on a mission to explore the different datasets available publicly, and hopefully to add to that list. One key reason I’m passionate about this is that data…

ML and IR Tomography

I studied Electrical and Computer Engineering at UCT, and the final year project was my chance to really dive deep into a topic. I chose IR tomography, and explored various…

Mapping Baobabs, Part 3 - Model Applicability

In the previous two posts, we built a Species Distribution Model and used it to predict the density of Baobab trees in Zimbabwe. Then we tried some more complex models…

Mapping Baobabs, Part 2 - Qualifying Model Performance and More Complex Models

Update, July 2020: GEE now has a good random forest implementation (ee.Classifier.smileRandomForest) that can do regression - I’d suggest using that instead of the approach…

Mapping Baobabs, Part 1 - Modelling the Density of Baobab Trees in Zimbabwe with a Linear Model in GEE

This is the first in a multi-part series exploring Species Distribution Modelling (SDM) with the Google Earth Engine. In this post, we’ll take a look at the data we’ll be…

Christmas games - simulation and lazy stats

This Christmas, I was introduced to several new games by my new extended family. Much fun was had making up new games and rediscovering old, but one game annoyed me…

Curious correlations

I wanted to write this up to show how easy it is becoming to test ideas and find interesting trends in data. Please don’t draw too many conclusions from the numbers here -…

init(self): What is this blog