Eleni's Garden

Upload the media to your S3 bucket

2024-08-09T12:12:12.290Z

Upload the media to your S3 bucket

Part of: Hosting Obsidian S3 media publicly

Make sure you have followed the previous steps:

Use the Obsidian Plugin

You can use the excellent S3 Image Uploader plugin to automatically upload images to your S3 bucket.

Installation

You can find the plugin in the list of Obsidian community plugins.

Configure credentials for upload

Provide the Access Key ID and Secret Access Key from the IAM you've previously created.
Provide your bucket's region (e.g. eu-central-1)
Provide your bucket's name (e.g. assets.example.com)

Configure Cloudfront URL

The final step is to configure the domain we previously setup:

Enable Use custom image URL
Set Custom Image URL to the domain you previously configured with Cloudfront (e.g. assets.example.com). For example for this blog:

Manual Alternative: Manage S3 bucket with Cyberduck 🦆

You can also manually manage S3 "files" (objects) using Cyberduck.
There's quite a bit of documentation, but there are suggestions I didn't like (e.g. user with full S3 permissions), and some parts that didn't work (e.g. bucket name in the server url).

Here's what did work for me:

Setup an IAM user to manage an S3 bucket
Set up an S3 (HTTPS) bookmark in Cyberduck:
1. Provide the Access Key ID and Secret Access Key.
  PS: if you forgot to write these down, you can recreate them.
2. Set Server to s3.amazonaws.com
3. Set More Options -> Path to the name of your bucket, e.g. /assets.example.com
That's it 🍰

Setup custom AWS CloudFront CNAME

2024-08-09T12:05:56.299Z

Setup custom AWS CloudFront CNAME

Part of: Hosting Obsidian S3 media publicly

AWS CloudFront distributions provide a random subdomain (e.g. a2cc2124adda.cloudfront.net), which comes with an SSL certificate.
If we want to use a custom domain name, we have to also setup an SSL certificate.

Here's the summary to save the trips to the various guides scattered around AWS docs.

Assuming we want to point an AWS CloudFront to use assets.example.com:

On AWS Certificate Manager (ACM)
- Create a new public certificate
- NB: The region must be US-East N. Virginia!
- You can also get it from somewhere else and import it
On the DNS provider:
- Add verification CNAME
- Add CNAME and pointing from domain to distribution domain, e.g. assets.example.com -> a2cc2124adda.cloudfront.net
Ensure it all works:

> dig assets.example.com
# ....
assets.example.com.	13104	IN	CNAME	a2cc2124adda.cloudfront.net.
# a2cc2124adda.cloudfront.net. 54 IN	A	3.162.78.124
# ....

On AWS CloudFront
- Find the distribution
- Settings -> Edit
- Set up the CNAME from the previous step, i.e. assets.example.com
- Set up the certificate created above
Done, your images should now be accessible through your prefered domain! 🍰

Setup an IAM user to manage an S3 bucket

2024-08-09T12:05:56.285Z

Setup an IAM user to manage an S3 bucket

Part of: Hosting Obsidian S3 media publicly

We can create an IAM user that only has permissions to manage a desired bucket, e.g. S3-assets-manager.

The most basic permissions would be to just list and manage objects in the entire bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::assets.example.com",
                "arn:aws:s3:::assets.example.com/*"
            ]
        }
    ]
}

[!info] More Permissions
Permission to list buckets may also be useful in some cases (e.g. if needed for an integration that's bucket-name agnostic).

Setup S3 bucket for use with Cloudfront

2024-08-09T12:05:56.280Z

Setup S3 bucket for use with Cloudfront

Part of: Hosting Obsidian S3 media publicly

Create an S3 bucket
e.g. assets.example.com

[!info] Public or Private?
Unless you have a specific public use in mind, you can keep this bucket private.
Create a Cloudfront distribution
- As Origin domain select your S3 bucket
- You can leave Custom SSL Certificate empty to start with and set it up later (see Setup custom AWS CloudFront CNAME)
Done 🍰

Welcome

2025-06-01T10:34:12.731Z

👋 'ello!

Thanks for stopping by 🙂
I'm Eleni Lixourioti (🔊) , welcome to my "digital garden"!

Find me on

🤖 Github | 💼 LinkedIn | 🦋 Bluesky
🤳 Instagram | 📷 Adobe Portfolio

...or by good old email

Pinned

Microbit Project 3 - Plants have feelings too

Recently added

Top tags

2024-03-05T15:30:33.743Z

Top tags

Topic	Count
#how-to	9
#python	7
#aws	5
#embedded	5
#microbit	5
#django	2
#outdated	2
#cycling	1
#dev	1
#elixir	1

The Time-Travelling Islands

2024-03-05T12:24:23.644Z

The Time-Travelling Islands

Original publication date: May 11, 2019

Writing about your experience on time handling in software as a developer seems to be the "write your own blog-engine" of blogging. So here goes 😀

Intro

One of features I've worked on in a previous project, involved producing personalised aggregations for things like sports performance and other personal data (ex. running, cycling, sleep patterns...). The particularity of this type of data was that it introduced a model of time ranges that don't only occur at a fixed point in time (at the User's time zone), but must always remain "relative" to our Users, wherever they might be.

To better describe this with an example: Say one morning you wake up and you go for a run, while in London. If the next month you move to New York, then your run should still be categorised as a "morning" run, since you did it between 6am-11am local time, even if, by New York standards, it would have been an "late night" run.

Similarly, if you cared about your sleep patterns, taking a log that says "you slept between 10pm and 6am" and converting it to another timezone, essentially makes it useless to you as a User. If we moved it to another timezone, then it could suddenly appear as if you were sleeping in the middle of the day. So the range is only relevant when paired with the timezone in which it occurred, without converting it to the current timezone.

Time for Motivation 🏃‍♀️

One of the applications for these aggregations was to run friendly competitions between users (this was a health-related platform, where maintaining motivation was a key target).

The competitions would typically have a daily, weekly or monthly duration, and included users world-wide 🌍

To take the example of a daily competition, we needed to observe the data of competitors on a single date, ex. 30/12/2011.

One immediate challenge we faced was that a "day" on its own, without a specific time associated with it, cannot have a single timezone. Sounds familiar? It's the same problem as describing what "morning" is! "Day", "morning", "month"... these time-ranges are always relative to our location.

As we wanted to handle data for a specific "day", before breaking the date down into time ranges for individual time zones, we first needed to answer another, more basic question:

When does a day start and end, world-wide?

So, what are the earliest and latest points that a date occurs, in UTC time?

Pacific Rim

One way to do that would be to find the earliest and latest timezone offsets, and use those as references. And we did exactly that. After a brief lookup, we found Pacific/Kiritimati (Kiribati, Line Islands 🇰🇮) as the earliest offset, with UTC+14 year-round, and Pacific/Apia (Apia, Samoa 🇼🇸), as the latest with a UTC-11 offset.

And it worked! In fact things worked quite smoothly for us for a while...

Happy New Year!

...until we came back from Christmas holidays, on 2nd Jan 2012.

Users were also back from holidays, and reporting issues on our support channels (what a great way to start you year 😭)!

When we checked some of our logs and databases we realised that some of our aggregate data had not been generated for the last 3 days!

After hours of tracking the problem, we realised that our date duration calculation was returning a wrong value. A day, it turns out, had now a duration of zero hours. How was that even possible?

Back to the Future

Most people don't usually read Time Zone News. If we had, we would have noticed the following headline:

Samoa jumps forward in time

Samoa will switch time zones by redrawing the international dateline. The change will occur at midnight on December 29, 2011, taking the Pacific island nation straight into December 31, 2011.

Turns out Samoa didn't want to be stuck in a timezone that's inconvenient for itself and its neighbours, so it decided to pull a Back To the Future manoeuvre, changing its timezone from UTC-11 to UTC+14 and literally skipping an entire day 🍻

Take aways

After quite a lot of digging and many "wtf" exclamations, we finally figured where the problem was coming from, quickly deployed a fix and no Time Lords needed to come and save us.

What could we have done better?

We could have had an automated regular task checking for updates on our time zone library (in our case that would have been pytz). Time zone changes are rarely decided and implemented in short notice (though that's not always true). Staying up to date definitely makes you safer.
Consider if your problem requires a time zone by its local name (ex. Europe/London), or if an explicit UTC offset (ex. UTC+1) is enough/appropriate. Most of the time you will need to account for things like DST, so the offset will not be enough and can change, so the general advice is to use and store time zone names. But in some cases, like ours, we could have hard-coded UTC+14 and UTC-11 without losing any information / risking breaking our product.
Read the news (not just Hacker News)! Our work as product makers doesn't exist in a separate bubble from the rest of the world, as much as we like to limit the scope of what we do for simplicity's sake. Many large news sources like the BBC had also reported on the change.

Reading

Full article on Samoa's timezone change
If you haven't read this yet: UTC is enough for everyone, right?

Phoenix Plug.BasicAuth config at runtime

2025-06-01T16:02:09.459Z

Phoenix Plug.BasicAuth config at runtime

HTTP Basic Auth a simple way of restricting access to an app, without introducing a full-blown user authentication system.

Luckily Plug comes with Plug.BasicAuth already built-in for us to use.

This is the usage example from the docs :

# lib/my_app.ex
import Plug.BasicAuth
plug :basic_auth, Application.compile_env(:my_app, :basic_auth)

# config/config.exs
config :my_app, :basic_auth, username: "hello", password: "secret"

Seems simple enough! In a real-life Phoenix app, we must probably:

add the plug to our Endpoint module, so we can protect all requests.
read the config from env variables, so we don't include secrets in our codebase files.

So something like:

# lib/my_app_web/endpoint.ex
defmodule MyAppWeb.Endpoint do
  # ...
  import Plug.BasicAuth, only: [basic_auth: 2]
  
  plug :basic_auth, Application.compile_env!(:my_app, :basic_auth)
  # ...
end

# config/config.exs
config :my_app, :basic_auth,
  username: System.get_env("AUTH_USERNAME", "user")
  password: System.get_env("AUTH_PASSWORD", "password")

The problem

But what happens if we do it like this and load the config in config.exs, as suggested?

This might not seem to matter much while developing locally, but, as soon as we want to build and deploy our app, it means that:

To change the config we need to recompile and redeploy our code
We need to have a way to make the secret variables available during build

These do sound like a bit of a pain...
👉 In fact, when it comes to env variables and external config, the elixir docs actively discourage using config.exs, in favour of runtime.exs, which allows us to read configuration during (surprise) runtime.

Let's fix this

So, can we just move our config into runtime.exs and be done?

Not right away! Our plug declaration is currently reading configuration at compile time. How do we know this? If you remember our Endpoint definition from above:

defmodule MyAppWeb.Endpoint do
  # ...
  plug :basic_auth, Application.compile_env!(:my_app, :basic_auth)
  # ...
end

We are using Application.compile_env!/2. This reads the application environment at compile time
And we are calling it in the module body. This is evaluated at compile time.

To fix this, we need to find alternatives for the two things above, so we can read the application environment at runtime:

Use a runtime-alternative to Application.compile_env!/2. E.g. Application.fetch_env!/2
Call within a function body, where it will be evaluated at runtime.

So how do we do this? One way to do this would be to make our own wrapper Plug around the built-in one.

We have two options here: as a function, or as a module.
Picking the simplest of the two, let's go with a function:

# lib/my_app_web/endpoint.ex
defmodule MyAppWeb.Endpoint do
  # ...
  plug :basic_auth

  # ...
  def basic_auth(conn, _opts) do
    basic_auth_config = Application.fetch_env!(:my_app, :basic_auth)
    Plug.BasicAuth.basic_auth(conn, basic_auth_config)
  end
end

We are now reading our config at runtime :)
Then we can just move our config to runtime.exs:

# config/runtime.exs
config :my_app, :basic_auth,
  username: System.get_env("AUTH_USERNAME", "user")
  password: System.get_env("AUTH_PASSWORD", "password")

That's it ✨

PS: If we then want to deploy our app to something like fly.io, we can then configure our secrets like any other secret:

fly secrets set AUTH_USERNAME=some-other-user AUTH_PASSWORD=very-secret-password

or use whatever equivalent env variable/secret management system.

Key takeaways

Be mindful of whether your code is reading configuration at compile time or at runtime.
Most of the time you can and should choose to read at runtime.
It's usually not too complicated to refactor your code to do so.
See also: Application docs on compile-time environment

Needle Alignment Tip - Singer Quantum Stylist 9960

2025-06-28T13:29:09.777Z

Your needle alignment is NOT broken - Singer Quantum Stylist 9960

While working on a project, I was doing something dodgy like trying to sew one million layers of fabric using an ancient needle. Not sure when the last time I cleaned my machine was. This resulted in the machine skipping and all the thread tangling up.

No worries, I untangled it all, cleaned up a bit, changed the needle, and so I was back on track. Only I wasn't: when I turned the machine back on, the needle would not align.

Things I tried that didn't work:

Turning it off and on again.
Changing the stitch setting
Worrying I broke the machine
Reading every manual on the internet
Watching every YouTube repair video where they take half the sewing machine apart and put it back together

The first time this happened, I decided to call it a day. Next day I set up the machine again, and was surprised to see everything worked.

Theory: it was definitely overheating.
Reality: it was overheating of my brain.

The second time this happened, I figured out what the actual problem was:
The needle's position was too low.

How? As I was manually moving the needle to slowly untangle things, I did not put the needle back to the top-most position.

🪡 If the needle is too low, the machine will not change the alignment.

This is presumably a protection mechanism, so that you don't change alignment while the needle is close to the foot, or through fabric.

⬆️ Takeaway 1: always set the needle to the top-most position.

Isn't this sewing machine 101? Maybe. Sometimes when using a digital sewing machine, it offers automations that you take for granted (e.g. the Singer Quantum Stylist 9960 has a ✂️ button that will reset the needle position and cut the thread for you).

🧠 Takeaway 2: just because you have a fancy machine, doesn't mean you can forget the basics.

Microbit Project 3 - Plants have feelings too

2024-03-05T12:52:27.121Z

Project # 3: Plants have feelings too

Original publication date: Dec. 30, 2015

I have recently been pretty bad at keeping my house plants alive. I need to add reminders to take care of my plants, and I usually end up ignoring them as "I'll do it when I get home", and then I almost never do. Wouldn't it be nice if the plant could tell you that it needs attention?

Plants don't have faces to express themselves, so what if the micro:bit could work for that purpose?

Excluding cacti, the one thing that plants usually need the most is ground moisture. How is moisture measured? The internet says that conductivity is a pretty good indicator.

So with a simple Ohmmeter we could measure the increase/decrease in resistance and by setting a few reasonable thresholds we can tell what the resistance corresponds to in terms of moisture levels.

Tools used:

3 cables with alligator clips
2 galvanised (I think) nails.
1 resistor (~1.2KΩ)

The nails were used as ground probes, one connected to power and one connected to pin0. The resistor was used as the "known" resistor for the ohmmeter. To replicate this you could use any resistor/ground probes/cabling. Just make sure the code is adapted for the resistor and that the probes are resistant to corrosion.

Also it goes without saying that you should keep the micro:bit at a safe distance from water and dirt ;)

In terms of programming, I set the micro:bit to display a happy, ok or sad face depending on the value of the resistor. Finding the correct thresholds was tricky as they can vary depending on the soil, distance between the probes etc. (code)

So, without further ado here's a watering demo of one of my plants (which I kept without water for a few days for that purpose as you can see from its horrible condition!):

As you can see with a bit of watering the plant becomes "OK" and with a bit more it becomes "Happy" ;)

A few potential extensions to this project (probably more important for outdoor plants...):

Light detector
Temperature detector

Microbit Project 2 - Snowflakes

2024-03-03T14:50:43.006Z

Project # 2: Snowflakes (almost)

Original publication date: Dec. 30, 2015

Since Christmas often means snow (at least for us living in the northern hemisphere) I thought to try to generate some snowflakes on the microbit.

Since snowflakes are symmetrical, we should only have to create a 3x3 corner and rotate that 45° at a time that to get the full shape. End result might resemble a star-wars spaceship instead of snowflakes. Equally seasonal?

Here's how it looked like:

and here's the code.

Microbit Project 1 - Dice roller

2024-03-03T14:50:19.062Z

Project # 1: Dice roller

Original publication date: Dec. 22, 2015

So the plan is to make a simple dice roller application.

The buttons should cycle through the various dice from a simple standard RPG poly set (d4, d6, d8, d10, d12, d20)
Once a die is selected the player should be able to roll it by shaking the micro:bit.

Possible difficulties:

Find a way to distinguish the rolling motion from other types of movement. My gut tells me this will be through detecting certain types of acceleration.

Continued...

I didn't get the chance to work on this yesterday due to Christmas preparations but today I got a couple of hours playing with the Micro:bit again :)

Back to detecting dice rolling, turns out this wasn't too hard but I did need to play a little bit with calibrating the thresholds for detecting consecutive changes in acceleration. There are probably more precise ways to do this, but it works for now!

I created a repo for the rest of the Micro:bit projects and this is the code for the dice roller.

I just realised this is very similar to what Martin O'Hanlon did with his 8-ball shuffle, our solution was actually pretty similar. I had to add some time thresholds as well, otherwise the shake event ended up happening way too easily and offset.

Meet Mei-Xiu

2024-03-03T10:41:01.061Z

Meet Mei-Xiu

Original publication date: Dec. 22, 2015

Sent to me by its previous host, Stewart Watkiss. Haven't flashed the device yet so it looks like the robot controlling hex is still loaded.

Environment setup

Started using upyed and Tom Viner's very convenient upyflashed. The tool was exiting with a trace in the event of no hex files being available, so I opened a PR to make this a bit better.

EDIT: Turns out my "fix" was skipping the sleep step which caused the loop cycles to be a bit too fast. Good thing Tom caught that 😉

It's Xmas

The Micro:bit gets a warm Christmas welcome by some of the other flashing LEDs in our house.

Karvonen HR training zones

2024-03-10T00:00:02.789Z

Karvonen HR training zones

Background

When training in activities like running or cycling, the concept of training "zones" is often used, in order to prescribe different levels of intensity.

There is value in training at different levels of intensity (low, moderate, aerobic, unaerobic etc), depending on what an athlete wants to achieve in a specific training session or longer term.

The general method

There are different ways of figuring out these zones. The most popular one, often also used by fitness programs and apps, relies on knowing one's maximum heart rate and with each zone being a percentage of it.

E.g. for someone with a max heart rate of 185bpm, training at 70% intensity would simply be:

185 * 0.7 = 129.5

This doesn't always work

Even when we have the max heart rate figured out, for some people these zones don't seem to correspond to how they actually experience different levels of intensity.

This may be due to them being highly trained athletes, well adapted to higher levels of intensity, or just regular physiological differences among individuals.

In my case, even at my fittest, I could never achieve the expected low intensity zones calculated with this method – my heart rate would always shoot past them at the slightest effort, even though I could sustain that effort almost indefinitely. What gives?

Alternative: the Karvonen method

The Karvonen method is an alternative, only slightly more elaborate method of calculating training zones.

This method calculates percentages based on heart rate reserve. In other words, takes both the max heart rate and the resting heart rate into account.

The full formula is defined as:

((Maximum HR – Resting HR) * Intensity) + Resting HR

So for someone with 185bpm max and 62bpm resting heart rate, to train at 70% intensity, we would have:

((185 - 62) * 0.7) + 62 = 148

This yields a very different result from the general method above (almost 20bmp more for the same intensity)!

While this is again a formula that can be imprecise, it seems to have worked well for me, providing much more realistic ranges that I can follow ✨ If you haven't stumbled upon it and you are frustrated with the default zones provided by your fitness app, it could be worth a try!

Calculator with sources

[!note] How do I find my max HR?
It's not always easy to calculate one's true maximum heart rate. For many people, a rough calculation based on their age (220 - age) may provide a good ballpark estimate.

Hosting Obsidian S3 media publicly

2024-08-09T12:06:35.032Z

Hosting Obsidian on media publicly

There are multiple options when we want to host media and larger files and serve them on the web.

Here's my setup for making any asset available through S3 (storage) and Cloudfront (CDN) on AWS, to take advantage of my existing account and the numerous tools and guides that already exist.

Public bucket? No thanks!

Many similar tutorials suggest allowing public read access to S3 buckets. This may not be a great idea. In this walkthrough we're going to use Cloudfront together with S3, to manage the content delivery. This means that the S3 bucket itself doesn't need to be public. Content is published only through Cloudfront.

Rough steps

Notes

GDPR on top of Django's model Collector

2024-03-05T12:16:52.390Z

Original publication date: Sep. 30 2018
Django meetup presentation

In the rise of GDPR many organisations have started implementing more and more features to deal with "housekeeping" of user data, compliance with new data processing/retention policies and explicit requests for data export or removal.

In practice, it can be hard to figure out all the possible connections of related data when given a specific user entity. In the context of complex relational databases, we are probably faced with inspecting large trees of relations and constraints.

Django tries to partially solve this problem by defining and managing most relations and constraints on the application level. Users of the Django admin may be familiar with this delete confirmation view:

This is possible because Django contains an object collector that will lookup those relations. When we run user.delete() Django will not just execute

DELETE FROM `myapp_users` WHERE `myapp_users`.`id` = x

but will first traverse all relations and collect all directly and indirectly related objects, which it will then clean up as appropriate. According to Django docs no ON DELETE CASCADE rule is ever defined in the DB, it is just emulated by the ORM.

While it can sometimes be frustrating to not have the DB fully manage those relations, we can, in some cases, use this Django feature to our advantage.

Collect data for export

Can we find out what data we have on this user?

This question pops up more and more frequently since the implementation of GDPR. The common use-case is when a user asks for an export of their data.

If we wanted to use the Django collector to get all related models for a given model, we could write something like:

from django.contrib.admin.utils import NestedObjects
  
def collect_user_data(user):
    collector = NestedObjects(using="replica")
    collector.collect([user])
    return collector.data

This would return all data related to our user instance (as long as the ORM knows about the relations).

This can be built upon for more complex cases:

from django.contrib.admin.utils import NestedObjects

# Imaginary module, used as an example
from myapp.housekeeping import collect, IP_ASSOCIATED_MODELS 


class CustomUserCollector(NestedObjects):
    def __init__(self, user, *args, **kwargs):
        self.user = user
        if 'using' not in kwargs:
            kwargs['using'] = "replica"
        super(CustomUserCollector, self).__init__(*args, **kwargs)

    def collect_from_db(self):
        self.collect([self.user])
        
        # Maybe we have some models that don't have a FK to user,
        # but are still somehow associated...
        for model in IP_ASSOCIATED_MODELS:
            self.collect(
                model.objects.filter(ip__in=self.user.known_ips))
    
    def collect_from_nonrel(self):
        # Maybe we have other types of stores in our system
        # that may contain more user data...
        self.data['couchdb'] = collect.from_couchdb(user)
        self.data['redis_temp'] = collect.from_redis(user)

    def collect_remote(self):
        # Maybe there is an internal API where we store user data
        # with its own export mechanism
        self.data['some_ext_service'] = collect.from_api(user)

    def collect_all():
        self.collect_from_db()
        self.collect_from_nonrel()
        self.collect_remote()


def collect_user_data(user):
    collector = CustomUserCollector(user)
    collector.collect_all()
    return collector.data

Report on deleted data

Can we keep track of how much data we're deleting?

So, we've been happily collecting and exporting data, but now it's time to delete some of it. Maybe we need to remove accounts after receiving deletion requests from users, or we may need to tidy up after data that's no longer used.

Deleted data is data we no longer need to really worry about (DISCLAIMER: I enjoy deleting stuff, including code).

If you simply remove an instance through the ORM, you will get back a summary of what was deleted. From the Django docs

Entry.objects.filter(pub_date__year=2005).delete()
# (5, {'webapp.Entry': 5})

Behind the scenes, django uses the same model collector to figure out what needs to be deleted and create a report.

A more complex deletion script could look something like:

def delete_all_the_things():
    inactive_users = fetch_inactive_users()
    inactive_users.delete()
    
    users_pending_delete = fetch_pending_deletes()  
    users_pending_delete.delete()
      
    for mailing_list in mailing_lists:
        unconfirmed_subs = fetch_unconfirmed_subs(mailing_list)
        unconfirmed_subs.delete()  

    # ...

etc.

If we wanted to check how many things we've just deleted, we would need to collect the output of each .delete() or log the individual statements.
We could also collect them and produce a summary, using an accumulator:

class DeleteStats(object):
    def __init__(self, delete_output=None):
        super(DeleteStats, self).__init__()
        self.total_removed = 0
        self.removed_by_model = defaultdict(int)

        if delete_output:
            self.extend(delete_output)

    def extend(self, delete_output):
        total_removed, removed_by_model = delete_output

        self.total_removed += total_removed
        for _model, _count in removed_by_model.iteritems():
            self.removed_by_model[_model] += _count

        return self

    def __getitem__(self, index):
        # Pretend to be a tuple to match how the django API returns deletes
        if index == 0:
            return self.total_removed
        elif index == 1:
            return self.removed_by_model
        raise IndexError()

also available as a Gist with mini-docs

import logging  
from myapp.housekeeping import DeleteStats

logger = logging.getLogger(__name__)

def delete_all_the_things(): 
    stats = DeleteStats()  
 
    inactive_users = fetch_inactive_users()
    stats.extend(inactive_users.delete())
    
    users_pending_delete = fetch_pending_deletes()  
    stats.extend(users_pending_delete.delete())
          
    for mailing_list in mailing_lists:
        unconfirmed_subs = fetch_unconfirmed_subs(mailing_list)
        stats.extend(unconfirmed_subs.delete())  
  
    # ...  
    total_deleted, removed_by_model = stats  

    logger.info("Deleted {} objects in total".format(total_removed))

    for model_name, count in removed_by_model.iteritems():
          logger.info("- {model_name} deleted: {count}".format(
                model_name=model_name, count=count))

IRL, instead of one long function, you could pass the accumulator into your cleanup functions, implement your own re-usable logging etc.

Gotchas

Performance. When the collector traverses and fetches all these relations, things can get pretty slow. On bulk-deletes, Django will detect if a model requires further cleanup of related objects and can end up iterating over each object marked for deletion (have a look at django.db.models.deletion.Collector#delete if you are curious to see how it works).
Over-relying on automated data discovery:
One size might not fit all (including all of your Django models in all their different shapes...).
Auto-discovered data may not be relevant. This sounds a bit silly, but imagine an export of:
```
{
    "id": 12345,
    "first_name": "Bob",
    "password": "pbkdf2_sha256$...rlP2U5X9BuFEGF6603k=",  
    ... 
}
```
The user does not need to know their password hash. They may not need to know meta data related to some UI setting, or denormalised data relevant only for performance enhancement. In addition:
Auto-discovered data may require more context. Imagine an export of:
```
{
  "id": 12345,
    "first_name": "Linda",
    "recent_actions": [
        "id": 5,
        "timestamp": 1538317086,
    ],  
    ... 
}
```
Without any additional context, action "id": 5 means absolutely nothing to the user or any 3rd party.

For both of the above, consider creating an explicit transformation/serialisation layer that can add or remove fields as appropriate (I've used Serializer from the DRF for that purpose in the past, probably worthy of its own post)

Finally, a one-off automated solution is not a replacement for solid process. New features will be added. New services will be deployed. Data will change form and move about. It is worth having a data-housekeeping checklist to reference in every change, to ensure that your automated processes stay up to date.

Django ORM - Order of query evaluation

2024-03-03T14:50:10.628Z

Django ORM - Order of query evaluation

Original publication date: Aug. 12, 2016

When working with an ORM it is often easy to forget that whatever you write is ultimately translated to SQL queries.

Consider this script:

bookings = task.contractor.bookings.all()
task.delete()
bookings.delete()

This will run with no errors.

As you would expecttask will be deleted, however the bookings will persist in the db! What's going on?

The code above will result in 2 SQL queries:

One that deletes the task based on the task id
One that deletes bookings that appear as a foreign key to the related contractor table, via the related tasks table filtered by... the same task id that was just deleted!

As the task is deleted in the 1st query, then the 2nd query will yield no results (as the specific task id will no longer exist).

For this to work we just need to reverse the order of operations:

bookings = task.contractor.bookings.all()
bookings.delete()
task.delete()

A week with micro-bit

2024-03-03T10:41:50.188Z

A week with the BBC micro:bit

Mei-Xiu (RIP) was a BBC micro:bit device running MicroPython, one of five devices that were sent out to travel the world and the python community.

I had the chance to play with it during the xmas period in 2015.

This is a series of small posts I made at the time:

Read the whole story behind the project or follow the entire trip at Micro World Tour.

Originally also posted on Thoughtstreams.