Planet Linux Australia

,

Russell CokerConsidering Convergence

What is Convergence

In 2013 Kyle Rankin (at the time Linux Journal columnist and CSO of Purism) wrote a Linux Journal article about Linux convergence [1] (which means using a phone and a dock to replace a desktop) featuring the Nokia N900 smart phone and a chroot environment on the Motorola Droid 4 Android phone. Both of them have very limited hardware even by the standards of the day and neither of which were systems I’d consider using all the time. None of the Android phones I used at that time were at all comparable to any sort of desktop system I’d want to use.

Hardware for Convergence – Comparing a Phone to a Laptop

The first hardware issue for convergence is docks and other accessories to attach a small computer to hardware designed for larger computers. Laptop docks have been around for decades and for decades I haven’t been using them because they have all been expensive and specific to a particular model of laptop. Having an expensive dock at home and an expensive dock at the office and then replacing them both when the laptop is replaced may work well for some people but wasn’t something I wanted to do. The USB-C interface supports data, power, and DisplayPort video over the same cable and now USB-C docks start at about $20 on eBay and dock functionality is built in to many new monitors. I can take a USB-C device to the office of any large company and know there’s a good chance that there will be a USB-C dock ready for me to use. The fact that USB-C is a standard feature for phones gives obvious potential for convergence.

The next issue is performance. The Passmark benchmark seems like a reasonable way to compare CPUs [2]. It may not be the best benchmark but it has an excellent set of published results for Intel and AMD CPUs. I ran that benchmark on my Librem5 [3] and got a result of 507 for the CPU score. At the end of 2017 I got a Thinkpad X301 [4] which rates 678 on Passmark. So the Librem5 has 3/4 the CPU power of a laptop that was OK for my use in 2018. Given that the X301 was about the minimum specs for a PC that I can use (for things other than serious compiles, running VMs, etc) the Librem 5 has 3/4 the CPU power, only 3G of RAM compared to 6G, and 32G of storage compared to 64G. Here is the Passmark page for my Librem5 [5]. As an aside my Libnrem5 is apparently 25% faster than the other results for the same CPU – did the Purism people do something to make their device faster than most?

For me the Librem5 would be at the very low end of what I would consider a usable desktop system. A friend’s N900 (like the one Kyle used) won’t complete the Passmark test apparently due to the “Extended Instructions (NEON)” test failing. But of the rest of the tests most of them gave a result that was well below 10% of the result from the Librem5 and only the “Compression” and “CPU Single Threaded” tests managed to exceed 1/4 the speed of the Librem5. One thing to note when considering the specs of phones vs desktop systems is that the MicroSD cards designed for use in dashcams and other continuous recording devices have TBW ratings that compare well to SSDs designed for use in PCs, so swap to a MicroSD card should work reasonably well and be significantly faster than the hard disks I was using for swap in 2013!

In 2013 I was using a Thinkpad T420 as my main system [6], it had 8G of RAM (the same as my current laptop) although I noted that 4G was slow but usable at the time. Basically it seems that the Librem5 was about the sort of hardware I could have used for convergence in 2013. But by today’s standards and with the need to drive 4K monitors etc it’s not that great.

The N900 hardware specs seem very similar to the Thinkpads I was using from 1998 to about 2003. However a device for convergence will usually do more things than a laptop (IE phone and camera functionality) and software had become significantly more bloated in 1998 to 2013 time period. A Linux desktop system performed reasonably with 32MB of RAM in 1998 but by 2013 even 2G was limiting.

Software Issues for Convergence

Jeremiah Foster (Director PureOS at Purism) wrote an interesting overview of some of the software issues of convergence [7]. One of the most obvious is that the best app design for a small screen is often very different from that for a large screen. Phone apps usually have a single window that shows a view of only one part of the data that is being worked on (EG an email program that shows a list of messages or the contents of a single message but not both). Desktop apps of any complexity will either have support for multiple windows for different data (EG two messages displayed in different windows) or a single window with multiple different types of data (EG message list and a single message). What we ideally want is all the important apps to support changing modes when the active display is changed to one of a different size/resolution. The Purism people are doing some really good work in this regard. But it is a large project that needs to involve a huge range of apps.

The next thing that needs to be addressed is the OS interface for managing apps and metadata. On a phone you swipe from one part of the screen to get a list of apps while on a desktop you will probably have a small section of a large monitor reserved for showing a window list. On a desktop you will typically have an app to manage a list of items copied to the clipboard while on Android and iOS there is AFAIK no standard way to do that (there is a selection of apps in the Google Play Store to do this sort of thing).

Purism has a blog post by Sebastian Krzyszkowiak about some of the development of the OS to make it work better for convergence and the status of getting it in Debian [8].

The limitations in phone hardware force changes to the software. Software needs to use less memory because phone RAM can’t be upgraded. The OS needs to be configured for low RAM use which includes technologies like the zram kernel memory compression feature.

Security

When mobile phones first came out they were used for less secret data. Loss of a phone was annoying and expensive but not a security problem. Now phone theft for the purpose of gaining access to resources stored on the phone is becoming a known crime, here is a news report about a thief stealing credit cards and phones to receive the SMS notifications from banks [9]. We should expect that trend to continue, stealing mobile devices for ssh keys, management tools for cloud services, etc is something we should expect to happen.

A problem with mobile phones in current use is that they have one login used for all access from trivial things done in low security environments (EG paying for public transport) to sensitive things done in more secure environments (EG online banking and healthcare). Some applications take extra precautions for this EG the Android app I use for online banking requires authentication before performing any operations. The Samsung version of Android has a system called Knox for running a separate secured workspace [10]. I don’t think that the Knox approach would work well for a full Linux desktop environment, but something that provides some similar features would be a really good idea. Also running apps in containers as much as possible would be a good security feature, this is done by default in Android and desktop OSs could benefit from it.

The Linux desktop security model of logging in to a single account and getting access to everything has been outdated for a long time, probably ever since single-user Linux systems became popular. We need to change this for many reasons and convergence just makes it more urgent.

Conclusion

I have become convinced that convergence is the way of the future. It has the potential to make transporting computers easier, purchasing cheaper (buy just a phone and not buy desktop and laptop systems), and access to data more convenient. The Librem5 doesn’t seem up to the task for my use due to being slow and having short battery life, the PinePhone Pro has more powerful hardware and allegedly has better battery life [11] so it might work for my needs. The PinePhone Pro probably won’t meet the desktop computing needs of most people, but hardware keeps getting faster and cheaper so eventually most people could have their computing needs satisfied with a phone.

The current state of software for convergence and for Linux desktop security needs some improvement. I have some experience with Linux security so this is something I can help work on.

To work on improving this I asked Linux Australia for a grant for me and a friend to get PinePhone Pro devices and a selection of accessories to go with them. Having both a Librem5 and a PinePhone Pro means that I can test software in different configurations which will make developing software easier. Also having a friend who’s working on similar things will help a lot, especially as he has some low level hardware skills that I lack.

Linux Australia awarded the grant and now the PinePhones are in transit. Hopefully I will have a PinePhone in a couple of weeks to start work on this.

,

Linux AustraliaCouncil Meeting May 24th 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Neill Cox (Secretary)
  • Sae Ra Germaine  (Council)
  • Marcus Herstik (Council)
  • Russell Stuart (Treasurer)
  • Jonathan Woithe (Council)

 

Meeting opened at 19:34 AEDT by Joel Addison and quorum was achieved.

Minutes taken by Neill Cox

 

2. Log of correspondence

  • Email from OSI re:  The vital role of Open Source maintainers facing the Cyber Resilience Act
  • Email from OSI re: Updating contacts for OSI’s Affiliate Program
  • Email from Kathy Reid re: FYI Response from Esther Payne on Mastodon in response to Bid Information Session – re safe conferences – Sae Ra has responded to Kathy, no further action required.
  • March Activity Statement has been lodged
  • FOSS4G SoTM Oceania 2023 Sponsorship Opportunity – council to discuss at next meeting.
  • Email from Business Events Perth – EOI for running Everything Open in Perth. Council will respond.
  • Email from Luke Carbis – insurance help for Sunshine Coast WordPress meetup group – Venue requires Public Liability Insurance with coverage of $20m AUD. The WordPress Community PBC only covers $2m (USD). As they are a sub committee they can use LA’s insurance

3. Items for discussion

  • Consultation period for Russel Coker’s grant application for the purchase of two PinePhones to support the development of a FOSS phone (“FOSS on Mobile Phones”) has ended.  See http://lists.linux.org.au/pipermail/grants/2023-April/000143.html .
    • Motion to approve the grant request as outlined on the list with the requirement that the recipients submit a talk proposal on their achievements to the next Everything Open conference.
    • Moved by: Sae Ra Seconded: Neill. Passed unanimously

 

4. Items for noting

5. Other business

  • Drupal sub committee update
    • Drupal South conference was last week, Wednesday and Thursday plus sprint on Friday. Despite earlier nervousness tickets sales improved at the last moment, hitting 168 which was around the mind point of the expected range. Tickets sales approximately 19k, having two conferences in the same financial year was a bit tough. Expecting a small profit once numbers are finalised which will be towards the end of the month.
    • Catering, A/V and professional identity were a little more expensive than expected, but the conference ran smoothly with no issues.
    • Positive feedback from attendees. Could possibly have seated more people for roughly the same cost. But the venue worked well for allowing face to face contacts. Sponsors were happy with their space. Keynotes went well, both from Kiwi speakers.
    • Talks were recorded and will be posted to YouTube.
    • A new event this year was the Splash Awards recognising four different categories. High production values (and a lot of volunteer work) for the award ceremony, which was well received by the community and appreciated by the winners. Followed immediately by a networking event.
    • Planning is underway for Sydney which is expected to be a larger event. Also planning a bit of a changeout for the organising committee. Expectation is around 250, possibly 300 or even 350, but need to be careful about over engineering the conference and hence overspending. Sponsors are coming forward with some ideas for adding to the format of the conference, more interactive demos/workshops of sponsor products rather than just static advertising.
    • There will be some sort of event run in parallel to the GovCMS event in Canberra, but with a distinct identity separate from GovCMS.
  • Admin team update
    • Meeting this Friday with FastMail to gauge what they can and can’t do. Certificate renewals next week. Conferences to use letsencrypt. ANU will need the admin team to move servers, which will involve downtime for the mirror as it’s a single machine, but the others should be snapshot and failover with no downtime. There is nothing too critical on the mirror and no downtime notification is necessary.

 

  • Joomla sub committee update
    • Not much to report, meeting times have changed. 15-20 people are attending regularly with a fairly stable core and a small number of new people who tend to come and go. Updates to  the constitution are still being worked on and will be sent to LA when finalised.
    • Possible event in November, but it is more likely that there will not be one this year. There will probably be some global events that the Australian community will be involved in. This will be an opportunity to market an Australian conference.
    • Suggested that the Joomla community consider applying for grant funding for community events.
    • The global Joomla community is looking at using github holopins to encourage involvement.
    • Joomla Australia uses meetup for organising events, but is looking at transitioning to other tools. Zoom / Office Hours using kumospace. Currently using a hybrid model with face to face meetings in Brisbane and other community members joining virtually.

 

  • PyCon AU sub committee update
    • Richard is unable to attend in person but has provided this update:
    • The CFP is completed, with 204 proposals submitted (including specialist track proposals). That was above the 176 proposals that the team had hoped for, which would have resulted in a 50% acceptance rate on 88 sessions (4 tracks x 3 days). It’s also almost exactly in line with Melbourne 2017, so it’s tracking with the team’s expectations of reductions in numbers (the running theory is proposals trend will be matched with tickets trend).
    • Key dates from here:
      • CFP open: Tuesday, 11 April
      • CFP close: Sunday, 14 May AoE
      • Program Review Phase 1 (anonymous): 22 May–4 Jun
      • Program Review Phase 2 (anonymous): 5 Jun–10 Jun
      • Specialist Track program selection due: 10 Jun
      • Main program selection:  11 Jun
      • Talk acceptances sent out: June
      • Schedule published: early July

 

  • Flounder sub committee update
    • No representative in attendance.

 

  • LUV – in future we will invite LUV to provide reports
    • Neill will contact Alexar to invite them to come to the next sub committee meeting

 

  • WordPress sub committee
    • Sydney meetup continues to grow 20 to 25 people each month. Wil culled the meetup subscription down from 5,000 to about 1,000 but has grown since to about 1,700. Meetup working well for the WordPress community. This is helped by WordPress promoting it when users log in to their site administration pages.
    • Outside of Sydney, meetup planning is progressing well in Perth, Brisbane, Sunshine Coast, Adelaide and possibly Hobart .
    • Wil has put feelers out for a WordCamp in Sydney and has had quite a few responses (12), much more interest than there was for an online Australia wide one. Wil plans to run an information session with a goal of holding an event in November 2024. WordCamp has strict requirements for cheap venues which constrains the possible venues. It will be a new organising team with Wil supplying experience.
    • Sae Ra suggested considering libraries for possible venues as well as universities.

The post Council Meeting May 24th 2023 – Minutes appeared first on Linux Australia.

,

Leon Brooksnew life for an old (FTX) PSU, improved life for one human

the LEDs on this 5m strip happen to emit light centred on a red that does unexpectedly helpful things to (and surprisingly deeply within) a human routinely exposed to it.

it has been soldered to a Molex connector, plugged into a TFX power supply from a (retired: the MoBo is cactus) Small Form Factor PC, the assorted PSU connectors (and loose end from the strip) have been taped over.

the LED strip cost $10.24 including postage, the rest cost $0, the PSU is running at 12½% of capacity, consumes less power than a laptop plug-pack despite running a fan.

trial runs begin today.



,

Simon LyallAudiobooks – April 2023

1493: Uncovering the New World Columbus Created by Charles C. Mann

Mostly covering the Colombian Exchange between the Old and New Worlds. With stories of goods, species and people going in both directions. 4/5

The Late Monsieur Gallet by Georges Simenon

A salesman is found shot dead. Maigret finds he was leading a double life, but to what end and why does the case not make sense? 3/5

The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company by Robert Iger

Covers his career before CEO up to his original retirement. Lots of interesting stores and some tips for managers. 4/5

A Man’s Head by Georges Simenon

Suspicious a convicted murderer is innocent Maigret arranges his escape and surveillance. One mysterious man directly challenges Maigret to solve the mysterious web of people around the case. 3/5

Dreams of other worlds: The Amazing Story of Unmanned Space Exploration by Chris Impey and Holly Henry

The story of 12 unmanned space missions. Not as detailed as I’d like, around half of each mission’s chapter goes off on tangents but still good. 3/5

The Hanged Man of Saint-Pholien by Georges Simenon

An unexpected suicide leads Maigret to a pretentious group of youths and another suicide ten years previous. 3/5

Lucky: How Joe Biden Barely Won the Presidency by Jonathan Allen and Amie Parnes

Covering the 2020 US Presidential primaries and election. Mostly from the Democratic side. Reasonably good but not quite up to the 2008/2012 books. 4/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

,

Linux AustraliaCouncil Meeting May 10th 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Russell Stuart (Treasurer)
  • Sae Ra Germaine  (Council)
  • Marcus Herstik (Council)
  • Jonathan Woithe (Council)

 

Apologies 

  • Neill Cox (Secretary)

 

Not Present

 

Meeting opened at 19:33 AEST by Joel and quorum was achieved.

Minutes taken by Wil Brown

 

2. Log of correspondence

  • 05/03/2023: DrupalSouth Wellington Sponsor Invoicing
  • 05/05/2023: PyConAU Subcommittee report
  • 08/05/2023: Gladstone Regional Council expression of interest for Everything Open 2024
  • 08/05/2023: Hong Phuc Dang from the FOSSASIA community requested to catch up with LA council members
  • 08/05/2023: Mike O’Connor expression of interest for Everything Open 2024 at Adelaide Convention Centre
  • 09/05/2023: Glenda Farrar request for Everything Open 2024 bid document.

 

3. Items for discussion

  • PyConAU Subcommittee Report
    • Not doing so well as per budget – adjusting to the lack of current demand.
    • Possibly struggling to get speaking slots – get some clarification on speaker submissions.
  • Hong Phuc Dang from the FOSSASIA community
    • Action: Wil – respond and offer a Zoom catchup.
  • Everything Open 2024 – expression of interests/bids
    • We have 2 expressions of interest: 
      • Adelaide – need to see a revised costing: Dates: possibly end of Jan/start of Feb
      • Gladstone – putting together some quotes for a bid. Dates: possibly early to mid-year.
      • LA – ensure both bidding parties are aware of each other. Action: Joel
    • LA to put together a bid document template for EO/type conferences. Action: Sae Ra & Joel.
  • Grant applications have been posted on the list
    • Waiting for community feedback.
    • Closing date next week – report back at the next council meeting.
    • Council to ask some clarifying questions. Action: Jonathan
      • Clarification of community inclusion and benefits
      • Technical goals for the project

4. Items for noting

  • Fastmail update
    • Steve, Sae Ra, and Nicola to meet next week with Fastmail to see what options they have to host mail servers.
    • Mailing lists system is probably not in scope.
    • Looking for better mail system stability and better spam filtering.
    • LA to complete a needs analysis, pros/cons of staying and moving away from internal mail system hosting.

5. Other business

  • Everything Open bids
    • Given the 2 bids, if both are feasible could offer one the 2025 slot.
    • Depends on if PyconAU holds a conf in Adelaide in 2025 – may push EO Adelaide bid out to the following year.
  • Mailing lists general discussion on usage: 
    • Re: grants and community feedback
    • Mailing lists don’t have a large volume of traffic
    • What alternatives are people using?
    • Possibly look at LA’s communication tech stack at a future date

The post Council Meeting May 10th 2023 – Minutes appeared first on Linux Australia.

,

Tim SerongLonghorn in a Sandbox

In my last post, I wrote about how I taught sesdev (originally a tool for deploying Ceph clusters on virtual machines) to deploy k3s, because I wanted a little sandbox in which I could break learn more about Kubernetes. It’s nice to be able to do a toy deployment locally, on a bunch of VMs, on my own hardware, in my home office, rather than paying to do it on someone else’s computer. Given the k3s thing worked, I figured the next step was to teach sesdev how to deploy Longhorn so I could break that learn more about that too.

Teaching sesdev to deploy Longhorn meant asking it to:

  • Install nfs-client, open-iscsi and e2fsprogs packages on all nodes.
  • Make an ext4 filesystem on /dev/vdb on all the nodes that have extra disks, then mount that on /var/lib/longhorn.
  • Use kubectl label node -l 'node-role.kubernetes.io/master!=true' node.longhorn.io/create-default-disk=true to ensure Longhorn does its storage thing only on the nodes that aren’t the k3s master.
  • Install Longhorn with Helm, because that will install the latest version by default vs. using kubectl where you always explicitly need to specify the version.
  • Create an ingress so the UI is exposed… from all nodes, via HTTP, with no authentication. Remember: this is a sandbox – please don’t do this sort of thing in production!

So, now I can do this:

> sesdev create k3s --deploy-longhorn
=== Creating deployment "k3s-longhorn" with the following configuration === 

Deployment-wide parameters (applicable to all VMs in deployment):

- deployment ID:    k3s-longhorn
- number of VMs:    5
- version:          k3s
- OS:               tumbleweed
- public network:   10.20.78.0/24 

Proceed with deployment (y=yes, n=no, d=show details) ? [y]: y

=== Running shell command ===
vagrant up --no-destroy-on-error --provision
Bringing machine 'master' up with 'libvirt' provider…
Bringing machine 'node1' up with 'libvirt' provider…
Bringing machine 'node2' up with 'libvirt' provider…
Bringing machine 'node3' up with 'libvirt' provider…
Bringing machine 'node4' up with 'libvirt' provider…

[... lots more log noise here - this takes several minutes... ]

=== Deployment Finished ===

You can login into the cluster with:

  $ sesdev ssh k3s-longhorn

Longhorn will now be deploying, which may take some time.
After logging into the cluster, try these:

  # kubectl get pods -n longhorn-system --watch
  # kubectl get pods -n longhorn-system

The Longhorn UI will be accessible via any cluster IP address
(see the kubectl -n longhorn-system get ingress output above).
Note that no authentication is required.

…and, after another minute or two, I can access the Longhorn UI and try creating some volumes. There’s a brief period while the UI pod is still starting where it just says “404 page not found”, and later after the UI is up, there’s still other pods coming online, so on the Volume screen in the Longhorn UI an error appears: “failed to get the parameters: failed to get target node ID: cannot find a node that is ready and has the default engine image longhornio/longhorn-engine:v1.4.1 deployed“. Rest assured this goes away in due course (it’s not impossible I’m suffering here from rural Tasmanian internet lag pulling container images). Anyway, with my five nodes – four of which have an 8GB virtual disk for use by Longhorn – I end up with a bit less than 22GB storage available:

21.5 GiB isn’t much, but remember this is a toy deployment running in VMs on my desktop Linux box

Now for the fun part. Longhorn is a distributed storage solution, so I thought it would be interesting to see how it handled a couple of types of failure. The following tests are somewhat arbitrary (I’m really just kicking the tyres randomly at this stage) but Longhorn did, I think, behave pretty well given what I did to it.

Volumes in Longhorn consist of replicas stored as sparse files on a regular filesystem on each storage node. The Longhorn documentation recommends using a dedicated disk rather than just having /var/lib/longhorn backed by the root filesystem, so that’s what sesdev does: /var/lib/longhorn is an ext4 filesystem mounted on /dev/vdb. Now, what happens to Longhorn if that underlying block device suffers some kind of horrible failure? To test that, I used the Longhorn UI to create a 2GB volume, then attached that to the master node:

The Longhorn UI helpfully tells me the volume replicas are on node3, node4 and node1

Then, I ssh’d to the master node and with my 2GB Longhorn volume attached, made a filesystem on it and created a little file:

> sesdev ssh k3s-longhorn
Have a lot of fun...

master:~ # cat /proc/partitions 
major minor  #blocks  name 
 253        0   44040192 vda
 253        1       2048 vda1
 253        2      20480 vda2
 253        3   44016623 vda3
   8        0    2097152 sda

master:~ # mkfs /dev/sda
mke2fs 1.46.5 (30-Dec-2021)
Discarding device blocks: done                            
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: 3709b21c-b9a2-41c1-a6dd-e449bdeb275b
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912
Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

master:~ # mount /dev/sda /mnt
master:~ # echo foo > /mnt/foo
master:~ # cat /mnt/foo
foo

Then I went and trashed the block device backing one of the replicas:

> sesdev ssh k3s-longhorn node3
Have a lot of fun...

node3:~ # ls /var/lib/longhorn
engine-binaries  longhorn-disk.cfg  lost+found  replicas  unix-domain-socket

node3:~ # dd if=/dev/urandom of=/dev/vdb bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.486205 s, 216 MB/s

node3:~ # ls /var/lib/longhorn

node3:~ # dmesg|tail -n1
[ 6544.197183] EXT4-fs error (device vdb): ext4_map_blocks:607: inode #393220: block 1607168: comm longhorn: lblock 0 mapped to illegal pblock 1607168 (length 1) 

At this point, the Longhorn UI still showed the volume as green (healthy, ready, scheduled). Then, back on the master node, I tried creating another file:

master:~ # echo bar > /mnt/bar
master:~ # cat /mnt/bar
bar

That’s fine so far, but suddenly the Longhorn UI noticed that something very bad had happened:

The volume is still usable, but one of the replicas has failed

Ultimately node3 was rebooted and ended up stalled with the console requesting the root password for maintenance:

Failed to mount /var/lib/longhorn – Can’t find ext4 filesystem

Meanwhile, Longhorn went and rebuilt a third replica on node2:

All green again!

…and the volume remained usable the entire time:

master:~ # echo baz > /mnt/baz
master:~ # ls /mnt
bar  baz  foo  lost+found

That’s perfect!

Looking at the Node screen we could see that node3 was still down:

There may be disk size errors with down nodes (4.87 TiB looks a lot like integer overflow to me)

That’s OK, I was able to fix node3. I logged in on the console and ran mkfs.ext4 /dev/vdb then brought the node back up again.The disk remained unschedulable, because Longhorn was still expecting the ‘old’ disk to be there (I assume based on the UUID stored in /var/lib/longhorn/longhorn-disk.cfg) and of course the ‘new’ disk is empty. So I used the Longhorn UI to disable scheduling for that ‘old’ disk, then deleted it. Shortly after, Longhorn recognised the ‘new’ disk mounted at /var/lib/longhorn and everything was back to green across the board.

So Longhorn recovered well from the backing store of one replica going bad. Next I thought I’d try to break it from the other end by running a volume out of space. What follows is possibly not a fair test, because what I did was create a single Longhorn volume larger than the underlying disks, then filled that up. In normal usage, I assume one would ensure there’s plenty of backing storage available to service multiple volumes, that individual volumes wouldn’t generally be expected to get more than a certain percentage full, and that some sort of monitoring and/or alerting would be in place to warn of disk pressure.

With four nodes, each with a single 8GB disk, and Longhorn apparently reserving 2.33GB by default on each disk, that means no Longhorn volume can physically store more than a bit over 5.5GB of data (see the Size column in the previous screenshot). Given that the default setting for Storage Over Provisioning Percentage is 200, we’re actually allowed to allocate up to a bit under 11GB.

So I went and created a 10GB volume, attached that to the master node, created a filesystem on it, and wrote a whole lot of zeros to it:

master:~ # mkfs.ext4 /dev/sda
mke2fs 1.46.5 (30-Dec-2021)
[...]

master:~ # mount /dev/sda /mnt
master:~ # df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda        9.8G   24K  9.3G   1% /mnt

master:~ # dd if=/dev/zero of=/mnt/big-lot-of-zeros bs=1M status=progress
2357198848 bytes (2.4 GB, 2.2 GiB) copied, 107 s, 22.0 MB/s

While that dd was running, I was able to see the used space of the replicas increasing in the Longhorn UI:

Those little green bars eventually turn yellow as the disks approach full

After a few more minutes, the dd stalled…

master:~ # dd if=/dev/zero of=/mnt/big-lot-of-zeros bs=1M status=progress
9039773696 bytes (9.0 GB, 8.4 GiB) copied, 478 s, 18.9 MB/s

…there was a lot of unpleasantness on the master node’s console…

So many I/O errors!

…the replicas became unschedulable due to lack of space…

This doesn’t look good

…and finally the volume faulted:

This really doesn’t look good

Now what?

It turns out that Longhorn will actually recover if we’re able to somehow expand the disks that store the replicas. This is probably a good argument for backing Longhorn with an LVM volume on each node in real world deployments, because then you could just add another disk and extend the volume onto it. In my case though, given it’s all VMs and virtual block devices, I can actually just enlarge those devices. For each node then, I:

  1. Shut it down
  2. Ran qemu-img resize /var/lib/libvirt/images/k3s-longhorn_$NODE-vdb.qcow2 +8G
  3. Started it back up again and ran resize2fs /dev/vdb to take advantage of the extra disk space.

After doing that to node1, Longhorn realised there was enough space there and brought node1’s replica of my 10GB volume back online. It also summarily discarded the other two replicas from the still-full disks on node2 and node3, which didn’t yet have enough free space to be useful:

One usable replica is better than three unusable replicas

As I repeated the virtual disk expansion on the other nodes, Longhorn happily went off and recreated the missing replicas:

Finally I could re-attach the volume to the master node, and have a look to see how many of my zeros were actually written to the volume:

master:~ # cat /proc/partitions 
major minor  #blocks  name
 254        0   44040192 vda
 254        1       2048 vda1
 254        2      20480 vda2
 254        3   44016623 vda3
   8        0   10485760 sda

master:~ # mount /dev/sda /mnt
master:~ # ls -l /mnt
total 7839764
-rw-r--r-- 1 root root 8027897856 May  3 04:41 big-lot-of-zeros
drwx------ 2 root root      16384 May  3 04:34 lost+found

Recall that dd claimed to have written 9039773696 bytes before it stalled when the volume faulted, so I guess that last gigabyte of zeros is lost in the aether. But, recall also that this isn’t really a fair test – one overprovisioned volume deliberately being quickly and deliberately filled to breaking point vs. a production deployment with (presumably) multiple volumes that don’t fill quite so fast, and where one is hopefully paying at least a little bit of attention to disk pressure as time goes by.

It’s worth noting that in a situation where there are multiple Longhorn volumes, assuming one disk or LVM volume per node, the replicas will all share the same underlying disks, and once those disks are full it seems all the Longhorn volumes backed by them will fault. Given multiple Longhorn volumes, one solution – rather than expanding the underlying disks – is simply to delete a volume or two if you can stand to lose the data, or maybe delete some snapshots (I didn’t try the latter yet). Once there’s enough free space, the remaining volumes will come back online. If you’re really worried about this failure mode, you could always just disable overprovisioning in the first place – whether this makes sense or not will really depend on your workloads and their data usage patterns.

All in all, like I said earlier, I think Longhorn behaved pretty well given what I did to it. Some more information in the event log could perhaps be beneficial though. In the UI I can see warnings from longhorn-node-controller e.g. “the disk default-disk-1cdbc4e904539d26(/var/lib/longhorn/) on the node node1 has 3879731200 available, but requires reserved 2505089433, minimal 25% to schedule more replicas” and warnings from longhorn-engine-controller e.g. “Detected replica overprovisioned-r-73d18ad6 (10.42.3.19:10000) in error“, but I couldn’t find anything really obvious like “Dude, your disks are totally full!”

Later, I found more detail in the engine manager logs after generating a support bundle ([…] level=error msg=”I/O error” error=”tcp://10.42.4.34:10000: write /host/var/lib/longhorn/replicas/overprovisioned-c3b9b547/volume-head-003.img: no space left on device”) so the error information is available – maybe it’s just a matter of learning where to look for it.

,

Russell CokerLinks April 2023

Cory Doctorow has an insightful article Gig Work is the Opposite of Steampunk [1] about the horrors that companies like Amazon are forcing on their employees.

Valerie Aurora and Leigh Honeywell wrote an insightful article about the al Capone theory of sexual harassment [2]. Why people who sexually harass others usually perform other anti-social activity that is also easier to prosecute.

The IEEE has an interesting article about using ML for parts of the CPU design process, both the technical issues and the controversy about competing ideas which is probably caused by sexism [3].

“Love and taxes are forever my heart” is a line from an anime dating sim game that prepares US taxes [4]. Unfortunately it was removed from Steam. The existence of the game is a weird social commentary and removing the game because you can’t have an anime hottie do taxes is bizarre but also understandable given liability issues. There’s no mention in the review of whether male hotties are available for people who prefer dating guys. As an aside my accountant looks like he is allergic to exercise…

The Killdozer Book web site (which has an invalid SSL certificate so you have to click on “advanced” in Chrome to get to the content) has an insightful article debunking some of the stories about the Killdozer [5]. He wasn’t some sort of hero of freedom, he was just a jerk who reneged on a deal hoping to get more money, thought that laws shouldn’t apply to him, and killed himself because of it.

Apparently some big tech companies are knowingly hiring people to not work unlike the usual large corporate case of unknowingly hiring people to not work [6]. Silicon Valley is a good TV show, and it’s apparently realistic.

Ron Garrett wrote in insightful blog post about theoretical attacks on Bitcoin and how Bitcoin could be used [7]. The conclusion is not good for Bitcoin.

Compiler Explorer is a program that shows how various C++ compilers produce assembly code for various architectures, this site hosts the main active instance [8]. There are other instances, here is an instance that produces code for the Ruzzian Elbrus architecture [9]. The Elbrus Wikipedia page is interesting [10]. Apparently the Ruzzians don’t want this information to be spread, LOL.

The Smithsonian Magazine has an interesting article about pet parrots being taught to video call each other [11]. Apparently parrots are social animals and can develop psychological problems if kept alone, so the video calls can be good for them. Also the owners had to monitor the chats to ensure that they played nicely together, just like play-dates for kids!

Phoronix has an amusing article about the drama regarding the AMD Spectral Chicken bit in the Linux kernel source [12].

This page listing bad free software licenses is amusing [13].

The ACS has an interesting article about how Samsung fakes photos of the moon and presumably could fake other photos of notable objects that don’t change [14]. The way that they proved the forgery was interesting.

Matt Palmerdev-dependencies and Rust's unused_crate_dependencies lint

I’m in the process of getting super-strict about the code quality of cretrit, the comparison-revealing encryption library that underlies the queryable encryption of the Enquo project. While I’m going to write a whole big thing about Rust linting in the future, I bumped across a rather gnarly problem that I thought was worth sharing separately. The problem, in short, is that the unused_crate_dependencies lint interacts badly with crates that are only needed for benchmarking, such as (in my case) criterion.

Rust has a whole bucketload of “lints” that can help your codebase adhere to certain standards, by warning (or exploding) if the problem is detected. The unused_crate_dependencies lint, as the name suggests, gets snippy when there’s a crate listed in your Cargo.toml that doesn’t appear to be used anywhere.

All well and good so far. However, while Rust has the ability to specify “crates needed for running the testsuite” (the [dev-dependencies] section of Cargo.toml) separately from “crates needed for actually using this thing” ([dependencies]), it doesn’t have a way to specify “crates needed for running the benchmarks”. That is a problem when you’re using something like criterion for benchmarking, because it doesn’t get refered to at all in your project code – it only gets used in the benchmarks.

When building your codebase for running the test suite, the compiler sees that you’ve specified criterion as one of your “testsuite dependencies”, but it never gets used in your testsuite. This causes the unused_crate_dependencies lint to lose its tiny mind, and make your build look ugly (or fail).

Thankfully, the solution is very simple, once you know the trick (this could be the unofficial theme song of the entire Rust ecosystem). You need to refer to the criterion crate somewhere in the code that gets built during the testsuite. The lint tells you most of what you need to do (like most things Rust, it tries hard to be helpful), but since it’s a development dependency, you need a little extra secret sauce.

All you need to do is add these two lines to the bottom of your src/lib.rs (or src/main.rs for a binary crate):

#[cfg(test)]
use criterion as _;

For the less Rust-literate, this means “when the build-time configuration flag test is set, import the criterion crate, but don’t, like, allow it to actually be referred to”. This is enough to make the lint believe that the dependency is being used, and making it only happen when the test build-time config flag is set avoids the ugliness of it trying to refer to the crate during regular builds (which would fail, because criterion is only a dev-dependency).

Simple? Yes. Did it take me a lot of skull-sweat to figure out? You betcha. That’s why I’m writing it down – even if everyone else already knows this, at least future Matt will find this post next time he trips over this problem.

,

Russell CokerWrite a blog post in the style of Russell Coker

Feeling a bit bored I asked ChatGPT “Write a blog post in the style of Russell Coker” and the result is in the section below. I don’t know if ChatGPT knows that the person asking the question is the same as the person being asked about. If a human had created that I’d be certain that “great computer scientist and writer” was an attempt at flattery, for a machine I’m not sure.

I have not written a single book, but I expect that in some alternate universe some version of me has written several. I don’t know if humans would describe my writing as being known for “clarity, precision, and depth”. I would not be surprised if “no-one else wrote about it so I guess I’m forced to read what he wrote” would be a more common response.

The actual “article” part doesn’t seem to be in my style at all. Firstly it’s very short at only 312 words, while I have written some short posts most of them are much longer. To find this out I did some MySQL queries to get the lengths of posts (I used this blog post as inspiration [1]). Note that multiple sequential spaces counts as multiple words.

# get post ID and word count
SELECT id, LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1 AS wordcount FROM wp_posts where post_status = 'publish' and post_type='post';
# get average word count
SELECT avg(LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1) FROM wp_posts where post_status = 'publish' and post_type='post';
# get the first posts by length
SELECT id, LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1 AS wordcount, post_content FROM wp_posts where post_status = 'publish' and post_type='post' ORDER BY wordcount limit 10;
# get a count of the posts less than 312 words
SELECT count(*) from wp_posts where (LENGTH(post_content) - LENGTH(REPLACE(REPLACE(REPLACE(REPLACE(post_content, "\r", ""), "\n", ""), "\t", ""), " ", "")) + 1) < 312 and post_status = 'publish' and post_type='post';
# get a count of all posts
select count(*) from wp_posts where post_status = 'publish' and post_type='post';

It turns out that there are 333/1521 posts that are less than 312 words and the average length is 665 words. Of the shortest posts a large portion were written before Twitter became popular and had the sort of unimportant content that Twitter is good for.

It seems ironic that ChatGPT says that I'm known for "depth" and then writes a short post allegedly in my style.

As for the actual content of the "post", it's not something that I greatly disagree with but it's definitely not something I'd write. If some manager or marketing person wrote something like this and asked my opinion I'd probably suppress a sigh and say it's OK for it's target market.

I think that free software is better in many ways and for many reasons and that separately from the issue of merit free software is important because of the ways that it is used. We could debate various issues related to the quality of free software vs proprietary software but even people who don't believe that free software is of high quality have to admit that software that runs most mobile phones and most web servers is important. As much as I dislike Windows most desktop PCs and laptops run it so I can't claim it to be unimportant. So my first disagreement with the content is the issue of importance vs quality.

One thing that ChatGPT is known for is weak arguments that appear convincing. "This diversity often leads to better software, as bugs are found and fixed faster, and new features are added more quickly" could be opposed by "the payment for proprietary software often leads to better software, as bugs are found and fixed faster, and new features are added more quickly". I think that any case where "feature of option A gives this list of good results" can be rewritten as "feature of the opposite of option A gives the same list of good results" and be equally convincing is a poor argument. Also any argument that seems to rest on "free software is all written by hobbyists" is ignoring the demands of users who need things done on a non-hobby timeframe, the many profitable companies that develop and support free software, and the employment status of a large portion of my Facebook friends (many of whom are paid for ~2000 hours a year of free software development and many more of whom have paid free software development as part of their work supporting the use of free software). Really the entire blog post it wrote seems to me like something a human might write if their boss assigned them to write about the benefits of free software, and that they would then write a new article saying the exact opposite if their boss asked them to.

Finally I prefer the term "free software" to "open source software". The difference doesn't seem important to some people, but any being that doesn't know the difference won't do well at impersonating me.

ChatGPT Result

Hello everyone, I'm ChatGPT, and today I'm going to try my best to write a blog post in the style of the great computer scientist and writer, Russell Coker.

Russell Coker is a renowned software engineer and writer, who is best known for his contributions to the Linux community, especially the Debian project. He has authored several books and articles on topics ranging from security to software engineering, and his writing style is known for its clarity, precision, and depth.

So, without further ado, let's dive into the topic of this blog post, which is the importance of open-source software.

Open-source software is software whose source code is available for anyone to inspect, modify, and distribute. This means that anyone can use, study, and improve the software, without any restrictions or fees. Open-source software has been around for a long time, but it has become more prevalent and important in recent years, thanks to the rise of the internet and the digital economy.

Open-source software has several benefits, both for individuals and for society as a whole. One of the most significant benefits is that it fosters innovation and collaboration. When software is open-source, it means that anyone can contribute to it, which leads to a diverse range of perspectives and ideas. This diversity often leads to better software, as bugs are found and fixed faster, and new features are added more quickly.

Another benefit of open-source software is that it is often more secure than proprietary software. Because the source code is available for anyone to inspect, vulnerabilities and security flaws can be found and fixed faster. This means that open-source software is often more reliable and trustworthy than proprietary software, which may have hidden backdoors or vulnerabilities that can be exploited by attackers.

Open-source software is also often more cost-effective than proprietary software. Because it is free to use and distribute, organizations can save money on licensing fees and other costs associated with proprietary software. This can be particularly important for non-profit organizations or small businesses that may not have the resources to invest in expensive software licenses.

In conclusion, open-source software is an important part of the digital landscape, and its benefits are numerous. Whether you are a software developer, a business owner, or an individual user, open-source software has something to offer. So, the next time you are looking for software to use, consider open-source options, and help support the collaborative and innovative spirit of the open-source community.

,

Linux AustraliaCall for Expressions of Interest for Conferences in 2024 and beyond

Dear community,

It was wonderful to be able to bring the open technologies communities together in person at Everything Open 2023 in Naarm (Melbourne) after a few years of online events. We had a wide range of excellent presentations from our speakers, with the videos of these now available on the Linux Australia Mirror and YouTube for everyone to catch up on the sessions they missed. We would like to thank everyone who participated in Everything Open 2023. The conference would not have been possible without the generous support of our sponsors, including our Emperor Penguin sponsors Google and IBM, as well as the dedicated team who came together to put on the event in a relatively short amount of time. We would also like to thank the countless hours dedicated to the conference by our Speakers, Volunteers, and Organising Team.

It is time to start planning the open technologies conference for 2024 and beyond. Linux Australia would like to hear from people who are interested in bringing the conference to their city, or being a part of the organising team for the conference. This is a little bit different to previous years where we have only asked for bids for the entire conference. These are still welcome and encouraged, however we recognise that forming a team and organising a bid is a large task, especially for those who have not been a part of a conference team before.

We are seeking expressions of interest in submitting a bid or being a part of an organising team for the 2024 and/or 2025 conferences. As mentioned earlier, you do not have to have a complete bid prepared – it is more important to us that you register your interest now, then we can work with each of the interested parties in more detail. Please send through your interest to the Linux Australia Council at council@linux.org.au.

We will be running a Conference Bid Information Session at 7:30pm AEST (UTC+10) on 8 May 2023. At this session Joel and Sae Ra will go through what is involved with running a conference, including what needs to be presented as part of a bid for a location. It is also an opportunity to see a bit more behind the scenes of what goes into planning and running a conference, drawing upon the experience Linux Australia has had over the past two decades.
We invite anyone who is interested to register to attend the session at https://us02web.zoom.us/meeting/register/tZwrd-yprzwrH9Qnj-Ftc5s2pamZ2xibLKDM.

Key Dates:
* Conference Bid Information Session – 8 May 2023
* Expressions of Interest for 2024 Conference – Due 28 May 2023
* Expressions of Interest for 2025 Conference – Due 25 June 2023

If you have any further questions, please contact Council via email or our website form and we will help where we can.

Kind regards,

Linux Australia Council

The post Call for Expressions of Interest for Conferences in 2024 and beyond appeared first on Linux Australia.

,

Linux AustraliaCouncil Meeting April 26th 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Neill Cox (Secretary)
  • Russell Stuart (Treasurer)
  • Sae Ra Germaine  (Council)
  • Jonathan Woithe (Council)

 

Apologies 

  • Marcus Herstik (Council)

 

Not Present

 

Meeting opened at 19:33 AEDT by Joel Addison and quorum was achieved.

Minutes taken by Neill Cox

 

2. Log of correspondence

  • Drupal South transactions
  • BAS Time – Joel is woking through the remaining items with Russell
  • Free Software Australia held an event on 20 Apr
  • EO 2024 Adelaide Proposal – Joel and Sae Ra have met with Mike and made some suggestion for dealing with the venue. Joel and Sae Ra have also cautioned Mike against trying to grow too big too soon.
  • Russell Coker’s LA Membership has expired and he has asked us to renew it. Wil has overridden the expiry. Jonathan will notify Russell and suggest a password reset.
  • ASIC Invoice due for payment – it has been paid.

3. Items for discussion

  • VALA Request for Video Hosting: (link redacted)
    • Note that Sae Ra is the current president of VALA.
    • VALA is currently hosting their current videos with their previous A/V provider who have now decided to charge a significant fee.
    • Another VALA member has suggested that their could be mutual benefits in VALA hosting their content at LA and in return assisting with metadata, indexing, setting up a PeerTube instance and lining content to the National EDeposit
    • VALA have ethical and commercial concerns that mean they do not want to host their content on YouTube.
    • PeerTube looks like an interesting platform for LA to explore, but VALA would be willing to consider alternatives.
    • Improving the ability to find things on the LA mirror would certainly be worthwhile.
    • Council will ask the admin/mirror (Steve Walsh initially) team to provide an opinion on the impact on our infrastructure, particularly regarding storage.
    • PeerTube is a separate issue, but something to be considered as a future option. Sae Ra will start the discussion with Steve.

 

  • Australian Computer Museum Society Grant Proposal(s)
    • We are currently waiting on ACMS to respond to Jonathon’s feedback on their proposals. We need to approve a membership request from Adrian Franulović

 

Subcommittee Update

  • 19:35 Drupal South – David Sparks
    • Drupal South happening in just over two weeks. Going pretty well. 80 talk submissions. Been able to curate a good programme with two local keynotes. Initial ticket sales were strong, but have hit the usual lull before the final week. Even without more tickets, and without final sponsors who have not yet committed then looking at  2k loss. 150 ticket sales so far. Biggest difference to the budget this year is a lack of midsize sponsors. The two top tier sold out quickly. Not helped by the conference being in the same financial year as the last one, and also pressure on spending for many companies.
    • Six months ago it looked like a hefty profit.
    • A few venue issues, but nothing major. The venue is no more expensive than the hotels were.
    • Looking at an Australian venue for 2024, but it’s been hard to nail down a venue.
    • One new thing this year is an awards ceremony with six categories and 20 entries. May collapse the categories down to 4.
    • Considering awards only tickets for local agencies.
    • Suggestion that Drupal South consider multi year sponsorships for the future.
    • Talks are being recorded. Focussing on capturing slides and audio. The team have an A/V person who will be handling recording.
    • Good sponsorship for the awards, so decent trophies and post conference coverage expected.
    • Considering a one day conference in Canberra in September.

 

  • 19:45 Admin Team – Steve Walsh
    • Not a lot to report. Some domain renewals due. It looks like the letsencrypt certificate are better supported by government agencies so it should be possible to use them for the conference websites. Certificates expire in June. linux.conf.au expires on 16 June.
    • Our footprint is not large enough to provide top notch spam/virus filtering. It would also perhaps be good to have email not on our infrastructure in case of an outage.
    • Sae Ra and Steve will investigate commercial options.
    • Servers at ANU need to be moved to different racks. That will be a multiple step process but still involve an outage.
    • The old cisco servers will be auctioned off on ebay. We are unlikely to get much for them, but they should sell. Could offer them to the community but as is – no options for more changes to config.

 

  • 19:55 PyConAU – Unable to attend due to late notice
    • Unfortunately Richard Jones couldn’t attend or find a replacement due to the late notice.

 

  • 20:05 Joomla – Patrick Jackson
    • We should reach out to the Joomla team to see if we can help. It looks like their next meeting may be on 20 May. Sae Ra will try to attend.

 

  • 20:15 Flounder –  Russell Coker
    • No show.

 

4. Items for noting

  • Insurance renewals for three of four policies have gone through at a similar price to last year. We’re now waiting on the Travel Insurance policy.
  • QA Session for EO 2024 will be held on Monday 8 May. An email will be sent out tomorrow requesting registration for people interested in running a conference.

5. Other business

  • None.

The post Council Meeting April 26th 2023 – Minutes appeared first on Linux Australia.

,

Matt PalmerRutie and Magnus, Two Good Ways to Build Ruby Extensions in Rust

I wrote the Ruby bindings for the Enquo Project, my attempt to bring queryable encryption to all databases, using the Rutie library. Recently, I’ve rewritten the bindings to use Magnus instead, and I thought I’d put down my thoughts about the whole situation.

The Story So Far

The Enquo Project core cryptography is all written in Rust, as seems to be the vogue these days. Rust is fast, safe, and easily interoperable with most of the rest of the modern software development ecosystem, making it a good choice as a language to implement the cryptographic primitives that Enquo needs, like Order-Revealing Encryption.

Of course, since not everyone writes their applications in Rust, we need to provide the functionality of the Enquo client in the languages that people do use, such as Ruby, Python, and so on. Since re-writing all that cryptographic code in a myriad of languages would be tedious and error-prone, we instead provide bindings to the “core” Rust code. These are just thin shims of code that translate the data types and function calls between Rust and the target language.

Shim in a Can
Wrong sort of shim, but canned language bindings would be handy

As I’m most familiar with Ruby and its development ecosystem (particularly Ruby on Rails), it was natural that I’d make Ruby bindings for Enquo as my first target. Rummaging around, it seemed that Rutie was a good library to use, so off I went.

What are Rutie and Magnus, Anyway?

Both libraries share the same goal: provide the ability to write some Rust code, run that through a compiler, and produce something that can be loaded by the Ruby interpreter and used just like any other Ruby class. They’re both fairly “high level” interfaces, trying to abstract away much of the gory details, and do a lot of the common “heavy lifting” that can make writing bindings fiddly and annoying. Things like mapping data types (like strings and integers) between Rust data types and the closest equivalents in Ruby.

This mapping never goes perfectly smoothly. For example, Ruby integers don’t have a fixed range of values they can represent – you can store a huge number like 2256 more-or-less as easily as you can the number 12. But Rust, being a lower-level language, only has a set of integer types that have fixed boundaries, like the u32 type, which can only store integers between zero and about four billion (232 - 1, to be precise).

There’s also lots of little things that need to be just right, also, like translating the different memory management approaches of the languages, and dealing with a myriad of fiddly little issues like passing arguments and return values in and out of method calls, helpers for defining classes and methods (and pointing to the correct Rust functions), and so on.

A mass of tangled pipes and valves
This is what I imagine it looks like inside these libraries
(Hervé Cozanet / Wikimedia Commons, CC-BY-SA)

All in all, these libraries are fairly significant pieces of work, and I’m mighty glad that someone else has taken on the job of building (and maintaining!) them.

So Why the Change?

Good question.

It’s important to say at the outset that there’s nothing particularly wrong with Rutie. I found using Rutie to be very straightforward, and the Ruby bindings came together very quickly and easily. If someone chose to use Rutie for their project, I’m sure they’d have a good experience.

What made me take the time to rewrite using Magnus was a set of a few tiny things, which together gave me enough of a shove to do the work.

Firstly, I’d had a hiccup with Rutie’s support of newer versions of Ruby, particularly 3.2 (PR). Also, I’d hit a couple of segfault issues, which were ultimately caused by Ruby garbage-collecting data out from underneath me. These were ultimately my fault, of course, but Rutie wasn’t helping me out in avoiding the problems in the first place.

Finally, while Rutie helped translate data types, there was still a bit of boilerplate and ugliness that needed to be included. This wasn’t a showstopper, but I’m appreciating the extra smoothness that Magnus provides here.

As an example, here’s what’s required in Rutie to get “native” Rust data types from Ruby method parameters (and the self reference to the current object):

fn enquo_field_decrypt_text(ciphertext_obj: RString, context_obj: RString) -> RString {
    let ciphertext = ciphertext_obj.to_str_unchecked();
    let context = context_obj.to_vec_u8_unchecked();

    let field = rbself.get_data(&*FIELD_WRAPPER);
    // etc etc etc

The equivalent in Magnus is just the function signature:

fn decrypt_text(&self, ciphertext: String, context: String) -> Result<String, magnus::Error> {

You can also see there that Magnus signals an exception via the Result return value, while Rutie’s approach to raising an exception involves poking the Ruby VM directly, which always struck me as a bit ugly.

There are several other minor things in Magnus (like its cleaner approach to wrapping structs so they can be stored in Ruby objects) that I’m appreciating, too. Never discount the power of ergonomics for making a happy developer.

The End Result

I spent a bit over half of last weekend doing the rewrite – maybe ten hours of so. Since Magnus did more type checking and data validation, and its approach to error handling was smoother, I took the opportunity to rewrite a bunch of Ruby “wrapper” code I’d written (which just existed to check things like ranges of values and string encodings) into Rust, as well.

To make sure that the conversion was accurate, I added a heap more unit tests to the bindings. I also took the opportunity to restructure the codebase to split the code for the different Ruby classes into separate files, which I hadn’t done initially as the code had originally accreted, rather than being purposefully written.

All up, though, my rewrite ended up removing over 60 lines (excluding the extra specs I added):

$ git diff --stat -- lib ext/enquo/src
 ruby/ext/enquo/src/field.rs       | 342 ++++++++++++++++++++++++++++++++++++++
 ruby/ext/enquo/src/lib.rs         | 338 ++++---------------------------------
 ruby/ext/enquo/src/root.rs        |  39 +++++
 ruby/ext/enquo/src/root_key.rs    |  67 ++++++++
 ruby/lib/enquo.rb                 |   6 +-
 ruby/lib/enquo/field.rb           | 173 -------------------
 ruby/lib/enquo/root.rb            |  28 ----
 ruby/lib/enquo/root_key.rb        |   1 -
 ruby/lib/enquo/root_key/static.rb |  27 ---
 9 files changed, 479 insertions(+), 542 deletions(-)

Considering that I was translating from a “higher level” language into a “lower level” one, the removal of so much code is quite remarkable. Magnus was able to automagically replace rather a lot of raise ArgumentError if something.isnt_right code in those .rb files.

So, in conclusion, if you, too, are building Ruby extensions in Rust, while Rutie is a solid choice (and you probably should stick with it if you’re already using it), I highly recommend giving Magnus a look for your next extension.

,

Linux AustraliaCouncil Meeting April 12th 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Neill Cox (Secretary)
  • Russell Stuart (Treasurer)
  • Sae Ra Germaine  (Council)
  • Marcus Herstik (Council)
  • Jonathan Woithe (Council)

 

Apologies 

 

Not Present

 

Meeting opened at 19:41 AEDT by Joel Addison  and quorum was achieved.

Minutes taken by Neill Cox

 

2. Log of correspondence

  • Mike O’Connor – LCA/EverythingOpen even for 2024
    • We’ve been sent possible venues for EO/LCA in Adelaide. We need to respond with guidance on the budget proposal.
    • Venue cost is too high, catering also and projected numbers are too optimistic.
    • Sae Ra and Joel will contact Mike and suggest alterations. Russell will send an initial email.
    • Possibly we could run an information session for people considering a bid. A skeleton budget could also be provided once a recent actual one has been sanitised.
  • Australian Computer Museum Society Grant Proposal
    • They seem a good fit, but we need a bit more information. Jonathan will respond and ask for the proposal to be fleshed out a bit so we can be sure it does match our intended use of grants.
  • OSI Affiliate directors elections 2023
    • Voting has actually closed for this election.

3. Items for discussion

  • KiwiPycon(15-17 sept) is currently scheduled close to PyconAU (August)
    • Possibly we should get the two groups to discuss timing and its impact on the conferences.
    • Neill has volunteered to be the liaison between the LA council and PyConAu
    • KiwiPycon is not auspiced by LA, but having the conferences so close together may impact both conferences.

4. Items for noting

  • The grants programme is now open and has been announced on the mailing list.
    • Can we move the details of the grants programme to the public area of the website while leaving the application form in the logged in area. Jonathon will action this.
  • Wil is working on the community handbook and social media manager position description but is unlikely to finish either soon.
    • Draft versions are available for council members to review.

5. Other business

  • Next meeting is a subcommittee meeting, Joel will need to explain to Neill how to organise one.
  • Council will put out a call for conference bids ASAP. Joel and Sae Ra will put out a formal request by early next week (18th) as part of that we will propose an information session for teams considering a bid.

The post Council Meeting April 12th 2023 – Minutes appeared first on Linux Australia.

,

Russell CokerBTRFS Rebuild Time

In February I replaced a Dell T320 server with a HP Z640 workstation for a home server/workstation [1]. The T320 has 8*3.5″ drive bays which I had used to put 3*4TB disks in a BTRFS RAID-10 array for 6TB of usable capacity. The Z640 has only 2*3.5″ bays and 4*2.5″ bays, so one option I could have taken was to buy a 4TB 2.5″ SSD and keep the same 3*4TB array as before. Instead I chose to use an 8TB disk I had spare in an array with one of the original 4TB disks and some extra on NVMe devices (the system has 2*1TB NVMe devices which are used as a 380G RAID-1 for the root filesystem and the rest for the storage array). It’s nice how BTRFS allows putting any storage you have in a RAID-10 configuration.

Unfortunately it seems that I chose the wrong 4TB disk to use for this as it failed three days ago. It gave thousands of read and write errors and Linux decided that the drive no longer existed. I tried rebooting the system to get it in the BTRFS array again but it failed again and failed so quickly that it wasn’t even possible to use the data on it as part of a RAID rebuild. So I removed that disk and put in one of the other 4TB disks.

As the array is comprised of an 8TB disk and 3 other devices that don’t add up to 8TB the layout is the 8TB disk having one copy of everything and the other devices having parts of it. So the rebuild process comprised of copying data from the 8TB disk to the 4TB disk. For a RAID array run in the manner of Linux software RAID the rebuild of a RAID-1 involves a linear copy of data which is the optimal case for hard disks, copying 4TB of data in that manner would have an average speed of a bit over 100MB/s and take about 11 hours. With BTRFS the source disk has to be updated for each block that is recreated so the process was bottlenecked on writing to the 8TB disk. It took 2 days 23 hours to complete. The process involved reading 3,478,031MB and writing 4,405,545MB. The system was live for the process and some cron jobs etc were writing to the array, but in the 12 hours since the rebuild completed the array has had 7,038MB written. So presumably during the rebuild time about 42G of actual data were written to the array and the other 4.3TB written to the 8TB disk were from the process of copying 3.5TB from it to another device. Iostat reported that there were 645.36 TPS for the duration of the rebuild which seems like a decent number for a hard drive, during the process iostat reported that the drive had 99%+ of IO capacity used for the duration.

While waiting for this to complete I wrote a blog post about storage trends [2]. One thing I didn’t mention in that post is that if you are the type of person who checks the rebuild process fifty times a day then that should be counted as part of the cost of using slow storage. If instead of an 8TB disk plus some SSD storage I had used 2*4TB disks and 1*4TB SSD as I had considered doing then instead of having 3.8TB on one device I would have had about 2.5TB and the reconstruct would have probably taken 2/3 of the time. If I had moved the array to 3*4TB SSDs then it would have taken a small fraction of the time.

One thing to note is that I made a mistake in this operation by removing the failed device instead of doing a “btrfs replace” operation which can be significantly faster. If I had correctly done this then I would have written a blog post about the rebuild taking 2 days or something, the issues of hard drives being slow and me compulsively checking the progress would still apply.

,

Simon LyallAudiobooks – March 2023

Pictures at a Revolution: Five Movies and the Birth of the New Hollywood by Mark Harris

The story of the conception, making and release of the 5 Best Picture Nominees at the 1968 Oscars. 4 Classic films and one Bomb. Great book, Highly recommend. 4/5

The Great North Road: London to Edinburgh – 11 Days, 2 Wheels and 1 Ancient Highway by Steve Silk

A mix of travel and history (a 100 year old guide is used and compared). A nice relaxing read. 3/5

Learn Like a Pro: Science-Based Tools to Become Better at Anything by Barbara Oakley

A good general book on learning and study techniques. 3/5

Firepower: How Weapons Shaped Warfare by Paul Lockhart

How weapons developments from 1300 to the end of WW2 shaped how wars were fought at the tactical and operation level. Very interesting, recommended. 4/5

Pietr the Latvian by Georges Simenon

The First Inspector Maigret detective story. A dead body and a live businessman are both match the description of a notorious fraudster. A short interesting listen. 3/5

Build: An Unorthodox Guide to Making Things Worth Making by Tony Fadell

The author talks about his career (at including helping create the ipod, iphone and Nest) and lessons about building products and companies. 4/5

The Crime at Lock 14 by Georges Simenon

A woman’s body is found in a stable near a canal and Inspector Maigret is called to investigate. Her upper-class but disreputable English husband is suspected. Lots of descriptions of Canal life 3/5

Steve Jobs by Walter Isaacson

An authorised (although with plenty of warts) biography of the Apple Founder and CEO. Well written, detailed and interesting. 4/5

The Farthest Shore by Ursula K. Le Guin

Throughout Earthsea magic is dying and people are sicking or going mad. Archmage Sparrowhawk and young Prince set out to investigate. A very real-feeling story. 4/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

,

Russell CokerStorage Trends 2023

It’s been 2 years since my last blog post about storage trends [1].

Minimum Storage <=2TB

In 2021 I stated that as MSY had 2TB disks for $72 and 2TB SSD for $245 it was barely worth considering a 2TB disk and anything less than 2TB wasn’t worth considering. Now for 2TB storage from MSY NVMe starts at $129, SATA SSD starts at $143, and hard disks start at $75. I guess that NVMe is slightly cheaper due to some combination of economies of scale for manufacture/sales and having less postage costs. It really doesn’t make sense to consider hard disks for storing 2TB or less.

For storage for a small system (PC or laptop) the cheapest storage device is $19 for a 128G SATA SSD. But it wouldn’t make sense to buy that when you can get a 256G SATA SSD for $22 or a 240G NVMe device for $23, saving $3 on storage wouldn’t make any sense. For 512G of storage the prices are $32 for NVMe and $33 for SATA SSD. For 1TB of storage the prices start at $68 for SATA SSD and $74 for NVMe. Probably for the vast majority of home users 1TB of SATA SSD or NVMe is the minimum storage capacity to consider, the $50 price difference isn’t much when considering the entire price of a PC or laptop and anything less than 1TB will run out quickly with modern use.

Larger Storage 4TB+

The price for 4TB of storage from MSY is NVMe starting at $349, SATA SSD starting at $369, and hard disks starting at $115. If you need 4TB of RAID-1 storage then it might be worth saving $470 and getting hard drives for a home user. For business use it wouldn’t make sense. Some laptops have two NVMe sockets so 8TB of storage (or 4TB of RAID-1) in a laptop would be interesting.

For 8TB of storage the MSY prices are SATA SSD for $739 and hard drives starting at $179. Probably hard drives are the best choice for most situations where there is a need to store 8TB or more of data. But the prices are low enough to make 8TB SSD something that can be considered for home use, it doesn’t seem that long ago that the 4TB hard drives I bought for my home server were almost that expensive.

Big Storage

MSY doesn’t have 8TB NVMe, such devices are on eBay for $1700 for regular M.2 NVMe and just under $1000 for U.2 (server hot-swap devices). So if you need more than 8TB of NVMe storage then probably buying a server with U.2 built in is the correct solution.

For home users who need more than 8TB of storage hard drives are a good solution. One issue is that the more affordable and larger drives use Shingled Magnetic Recording (SMR) which has some different performance characteristics for certain workloads. Apparently SMR performs badly for anything other than large file storage.

Why MSY?

I primarily used MSY prices for this post because they are a reliable local store that has a list of prices that is easy to read. For everything in this post I can get better prices by using eBay, the StaticIce.com.au price comparison site [2], and the computing section of the OzBargain site that gamifies finding good prices [3]. But a good shopping strategy nowadays is to compare prices in a store to determine what items are in your price range and then shop around for price on the item you want. Checking all the different bargain sites for all these items would take much more time than I want to spend writing a blog post!

Conclusion

Hard drives don’t make sense for the vast majority of systems. Not for laptops, not for typical desktop PCs, and not for small business servers (say 8TB or less of RAID storage). Hard drives only make sense for dozens or hundreds of TB of storage and even then finding out how to deal with SMR issues is going to increase the pain of deployment. Maybe using a combination of SSD and hard drives to deal with the SMR issues is going to be a competitive advantage for NAS vendors in future.

NVMe looks like it’s on the way to being cheaper than SATA SSD. There is likely to be a good market for systems with NVMe as the only internal storage option.

The long term trend of systems without DVD drives and with maybe 2.5″ SATA devices but no 3.5″ SATA devices seems to lead to the GPU being the major part that needs to fit into a PC case that determines the overall size. Maybe there will be a new trend of GPUs connected to riser cards so they can be parallel to the motherboard for compact PCs.

For business desktop systems (IE low powered graphics hardware as it’s not for gaming) I expect that the trend will be towards NUC type devices which are already based around the M.2 as a storage device size.

,

Matt PalmerDatabase Encryption: If It's So Good, Why Isn't Everyone Doing It?

a wordcloud of organisations who have been reported to have had data breaches in 2022
Just some of the organisations that leaked data in 2022

It seems like just about every day there’s another report of another company getting “hacked” and having its sensitive data (or, worse, the sensitive data of its customers) stolen. Sometimes, people’s most intimate information gets dumped for the world to see. Other times it’s “just” used for identity theft, extortion, and other crimes. In the least worst case, the attacker gets cold feet, but people suffer stress and inconvenience from having to replace identity documents.

A great way to protect information from being leaked is to encrypt it. We encrypt data while it’s being sent over the Internet (with TLS), and we encrypt it when it’s “at rest” (with disk or volume encryption). Yet, everyone’s data seems to still get stolen on a regular basis. Why?

Because the data is kept online in an unencrypted form, sitting in the database while its being used. This means that attackers can just connect to the database, or trick the application into dumping the database, and all the data is just lying there, waiting to be misused.

It’s Not the Devs’ Fault, Though

You may be thinking that leaving an entire database full of sensitive data unencrypted seems like a terrible idea. And you’re right: it is a terrible idea. But it’s seemingly unavoidable.

The problem is that in order to do what a database does best (query, sort, and aggregate data), it needs to be able to know what the data is. When you encrypt data, however, all the database sees is a locked box.

a locked box
Not very useful for a database

The database can’t tell what’s in the locked box – whether it’s a number equal to 42, or a date that’s less than 2023-01-01, or a string that contains the substring “foo”. Every value is just an opaque blob of “stuff”, and the database is rendered completely useless.

Since modern applications usually rely pretty heavily on their database, it’s essentially impossible to build an application if you’ve turned your database into a glorified flat-file by encrypting everything in it. Thus, it’s hardly surprising that developers have to leave the data laying around unencrypted, for anyone to come along and take.

Introducing Enquo

I said before that having data unencrypted in a database is seemingly unavoidable. That’s because there are some innovative cryptographic techniques that can make it possible to query encrypted data.

Andy Dwyer being amazed
Indeed

The purpose of the Enquo project is to provide a common set of cryptographic primitives that implement ENcrypted QUery Operations (ie “Enquo”), and integrate those operations into databases, ORMs, and anywhere else that could benefit. The end goal is to provide the ability to encrypt all the data stored in any database server, while still allowing the data to be queried and aggregated.

So far, the project consists of these components:

  • the enquo-core library, that implements queryable encrypted integers, dates, and text in Rust and Ruby;
  • a PostgreSQL extension, pg_enquo, that allows PostgreSQL to query encrypted data; and
  • a Rails ActiveRecord extension, ActiveEnquo, that augments ActiveRecord to do the encryption/decryption required.

Support for other languages and ORMs is designed to be as straightforward as possible, and integration with other databases is mostly dependent on their own extensibility.

The project’s core tenets emphasise both uncompromising security, and a friendly developer experience.

Naturally, all Enquo code is open source, released under the MIT licence.

Would You Like To Know More?

Desire to know more intensifies
Everyone who uses a database...

If all this sounds relevant to your interests:

  1. If you use Ruby on Rails and PostgreSQL, you’re halfway home already. Follow the ActiveEnquo getting started tutorial and see how much of your data Enquo can already protect. When you find data you want to encrypt but can’t, tell me about it.

    • If you use Ruby and PostgreSQL with another ORM, such as Sequel, writing a plugin to support Enquo shouldn’t be too difficult. The ActiveEnquo code should give you a good start. If you get stuck, get in touch.
  2. If you use PostgreSQL with another programming language, tell me what language you use and we’ll work together to get bindings for that library created.

  3. If you use another database server, support is coming for your database of choice eventually, but at present there’s no timeline on support. On the off chance that you happen to be a hard-core database hacking expert, and would like to work on getting Enquo support in your preferred database server, I’d love to talk to you.

,

Lev LafayetteCOMP90024: Cluster and Cloud Computing For 2023

For the past few years, I have delivered some guest lectures and training for the University of Melbourne master's level course Cluster and Cloud Computing. This year's contribution has been expanded, which is not surprising as the course is apparently required for data science students as well as computer science students. Thus, for 2023 four presentations were given, with the workshop repeated three times! The first two presentations were an introduction to the Linux command line, followed by slightly more advanced content which included an introduction to shell scripting. The third lecture was the main presentation on Supercomputing and the Spartan HPC system in particular. The third presentation was a workshop on HPC job submission and and introduction to OpenMP and MPI programming with a concentration on using MPI4Py.

,

Russell CokerLinks March 2023

Interesting paper about a plan for eugenics in dogs with an aim to get human equivalent IQ within 100 generations [1]. It gets a bit silly when the author predicts IQs of 8000+ as there will eventually be limits of what can fit in one head. But the basic concept is good.

Interesting article about what happens inside a proton [2]. This makes some aspects of the Trisolar series and the Dragon’s Egg series seem less implausible.

Insightful article about how crypto-currencies really work [3]. Basically the vast majority of users trust some company that’s outside the scope of most financial regulations to act as their bank. Surprisingly the author doesn’t seem to identify such things as a Ponzi scheme.

Bruce Schneier wrote an interesting blog post about AIs as hackers [4].

Cory Doctorow wrote an insightful article titled “The ‘Enshittification’ of TikTok” which is about the enshittification of commercial Internet platforms in general [5]. We need more regulation of such things.

Cat Valente wrote an insightful article titled “Stop Talking to Each Other and Start Buying Things: Three Decades of Survival in the Desert of Social Media” about the desire to profit from social media repeatedly destroying platforms [6].

This Onion video has a good point, I don’t want to watch videos on news sites etc [7]. We need ad-blockers that can block video on all sites other than YouTube etc.

Wired has an interesting article about the machines that still need floppy disks, including early versions of the 747 [8]. There are devices to convert the floppy drive interface to a USB storage device which are being used on some systems but which presumably aren’t certified for a 747. The article says that 3.5″ disks cost $1 each because they are rare – that’s still cheaper than when they were first released.

Android Police has an interesting article about un-redacting information in PNG files [9]. It seems that some software on Pixel devices hasn’t been truncating files when editing them, just writing the new data over top and some platforms (notably Discord) send the entire file wuthout parsing it (unlike Twitter for example which removes EXIF data to protect users). Then even though a PNG file is compressed from the later part of the data someone can deduce the earlier data.

Teen Vogue has an insightful article about the harm that “influencer parents” do to their children [10].

Jonathan McDowell wrote a very informative blog post about his new RISC-V computer running Debian [11]. He says that it takes 10 hours to do a full Debian kernel build (compared to 14 minutes for my 18 core E5-2696) so it’s about 2% the CPU speed of a high end 2015 server CPU which is pretty good for an embedded devivce. That is similar to some of the low end Thinkpads that were on sale in 2015.

The Surviving Tomorrow site has an interesting article about a community where all property is community owned [12]. It’s an extremist Christian group and the article is written by a slightly different Christian extremist, but the organisation is interesting. A technology positive atheist versions of this would be good.

Bruce Schneier and Nathan E. Sanders co-wrote an insightful article about how AI could exploit the process of making laws [13]. We really need to crack down on political lobbying, any time a constitution is being amender prohibiting lobbying should be included.

Anarcat wrote a very informative blog post about the Framework laptops that are designed to be upgraded by the user [14]. The motherboard can be replaced and there are cases designed so you can use the old laptop motherboard as an embedded PC. Before 2017 I would have been very interested in such a laptop. Now I’ve moved to low power laptops and servers for serious compiles and a second-hand Thinkpad X1 Carbon costs less than a new Framework motherboard. But this will be a really good product for people with more demanding needs than mine. Pity they don’t have a keyboard with the Thinkpad Trackpoint.

,

Linux AustraliaCouncil Meeting 29th March 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Neill Cox (Secretary)
  • Russell Stuart (Treasurer)
  • Sae Ra Germaine  (Council)
  • Marcus Herstik (Council)
  • Jonathan Woithe (Council)

 

Apologies 

 

Not Present

 

Meeting opened at 20:09 AEDT by Joel Addison and a quorum was achieved.

Minutes taken by Neill Cox

 

2. Log of correspondence

  • Email: Kathy Reid: Update on LA’s new Mastodon account, changes to website and a WordPress issue to be followed up
  • Working With Children Checks for Victoria State
  • Westpac bank statements
  • PO Box renewal slip – has been paid
  • Number Resource Society (NRS) election cover letter and brochure – no action required
  • Email: Call for OSI board nomination – time for action has passed.
  • Email: Joomla Australia – Linux Australia Subcommittee Proposal – Joel has responded, and will follow up.
  • Email: DrupalSouth Wellington – Budget and Onboarding – Russell has responded
  • Email: Drupal South payments – Russell has responded
  • Email: Transaction Approval and Domestic Payments Limit (Drupal South)- Russell has responded
  • Email: PyCon AU starting funds – Russell has responded. Joel has also spoken to Russell Keith McGee in person about Everything Open’s experience with late registrations.

 

3. Items for discussion

  • LA website. Avada theme needs a licence to update for security patches and the latest version. $USD 69.
    • Motion by Joel, seconded by Wil, that we sign up for renewal. Carried unanimously. Russell will set up an automatic payment with the Treasurer’s CC
  • Appoint a Social Media Manager either within the council or externally and develop a social media/content marketing plan. Should be a remit of the Media & Comms Team? (from Kathy)
    • One possibility is that the social media manager could use WP’s ability to push content to social media.
    • Wil suggests that having someone fulfilling that role would be very useful.
    • Jonathan suggests that this sounds like a communications manager. Sae Ra notes that getting a single person to manage social media and communications is a lot of work for a single person.
    • Official communications should still be council led.
    • Currently the Media and Comms team only has one member, with part time help from a few others.
    • Council to put a call out for volunteers.
    • Sae Ra has been working on a comms toolkit for VALA which she could share. Worth including a style guide and instructions for preparing images etc for the website.
    • Wil to prepare an outline for the council to discuss. Jonathan to assist.
  • Grant Program – Joel proposes to open $10k of standard grants, $5k of community meetup grants (eg. small grants to LUGs and other open technologies meetups associated with LA to host events, helping to kick start local communities again)
    • Now that we are in a new year with a new council we need to open a new Grants program.
    • Wil thinks this is a good opportunity to provide guidance to LUGs on how to reboot. Advice on venues, getting speakers/topics etc.
    • Russell asks if we paid out any money during Everything Open. We did, just under $1,000. Russell also notes that the budget contains $35,000 for the grants program.
    • Jonathan suggests that we should not lock ourselves in too much, but allow some flexibility to allocate money in response to the popularity of the categories.
    • Marcus suggests avoiding specifying amounts for categories at all and just ask for proposals. The counter argument is that it’s useful to provide some guidance to applicants.
    • Sae Ra suggests including some showcases/examples of previous grants.
    • The expectation is that grants will be small, requests for large amounts will need extraordinary justification.
    • Proposal to extend the closing of the grants program to later in the year. This would mean that the grants program will no longer align with our financial year, and possibly annoy the next council.
    • We should set a calendar reminder to prompt people to get their proposals in before the program closes.
    • Motion by Joel, Seconded by Wil: Open $10,000 of community grants and $10,000 of standard grants. Carried unanimously.
  • Employee Assistance Program – Motion by Joel to approve funding to sign up for EAP as per budget, to give Council members access to an Employee Assistance Program.
    • The budget amount is $1,500 for 12 months.
    • Once we sign up details on how to access the program will be provided.
    • Motion by Joel that Council approves funding for the EAP. Seconded:  Marcus. Carried unanimously.

4. Items for noting

  • BAS report is due soon. Outstanding reconciliation need to be done.
  • Mailman is not working at the moment, admin team has been notified (by email which may not have worked). Sae Ra will make sure the admin team are aware.
  • Everything Open budget needs some work. Currently showing a $25,000 loss, mainly due to outstanding sponsorship. We do expect all of the remaining sponsors to pay. The EO budget should break even or make a small profit, but may not cover the LA tax.
  • Joel has sorted out Ariba now, which should reduce the pain of dealing with sponsors who use that system.
  • Marcus asked about who uses the LA Conference SSO system. Currently the only conferences using it are LCA and Everything Open, but it is available to any conference that runs under LA’s auspice. We should consider a change to the domain used for the SSO system.
  • There was a discussion about the future of LCA and EO. It was pointed out that the overlap between the conferences can be confusing. EO is an attempt to recognise the broader range of topics that LCA has been covering. Council’s official position is still that we will evaluate bids put forward and if the bid looks viable we will try to facilitate running it. Council takes no official position on whether future conferences will be LCA or EO.
  • Council will try to improve future communication around running conferences.

5. Other business

  • None

The post Council Meeting 29th March 2023 – Minutes appeared first on Linux Australia.

,

Michael StillHolman CLXRGB60 RGB WiFi garden light controllers and tasmota

Today I went forth to Bunnings in the rain to purchase a Holman CLXRGB60 RGB garden light controller so that I too could have fancy lighting in my garden and impress all those guests I never have over. I had been given hope by the Blakadder site that I would be able to flash tasmota onto the controller so it integrated with my Home Assistant home automation.

Unfortunately, it was not to be. Despite the device being TYWE3L based, the warning on the blakaddr site was correct, and this is a next-gen Tuya device where the crypto hasn’t been broken yet. Then again, I couldn’t even get this device to pair in the Holman app, so it clearly hates me.

This unfortunately means the excellent instructions from Jon Oxer were unforunately not helpful today. I think there is a theoretical option here to flash using the serial pins on the board, ala this guide. Also, it means my hair got wet for nothing.

So as to take revenge for my wet hair I have decided to pivot. The Holman lights seem quite well made, but they’re just 12 volt RGB PWM devices. So I can use their lights and build my own controller — although I need to ponder how to drive high current PWM I suppose. I thought it would therefore be useful to document the pinout on the Holman connector before I return the controller.

Labelled RGB Holman Garden Light 12v pinout

I hope this is helpful to someone else as well. I wonder what this connector is called?

,

Tim SerongTeaching an odd dog new tricks

We – that is to say the storage team at SUSE – have a tool we’ve been using for the past few years to help with development and testing of Ceph on SUSE Linux. It’s called sesdev because it was created largely for SES (SUSE Enterprise Storage) development. It’s essentially a wrapper around vagrant and libvirt that will spin up clusters of VMs running openSUSE or SLES, then deploy Ceph on them. You would never use such clusters in production, but it’s really nice to be able to easily spin up a cluster for testing purposes that behaves something like a real cluster would, then throw it away when you’re done.

I’ve recently been trying to spend more time playing with Kubernetes, which means I wanted to be able to spin up clusters of VMs running openSUSE or SLES, then deploy Kubernetes on them, then throw the clusters away when I was done, or when I broke something horribly and wanted to start over. Yes, I know there’s a bunch of other tools for doing toy Kubernetes deployments (minikube comes to mind), but given I already had sesdev and was pretty familiar with it, I thought it’d be worthwhile seeing if I could teach it to deploy k3s, a particularly lightweight version of Kubernetes. Turns out that wasn’t too difficult, so now I can do this:

> sesdev create k3s
=== Creating deployment "k3s" with the following configuration === 
Deployment-wide parameters (applicable to all VMs in deployment):
deployment ID:    k3s
number of VMs:    5
version:          k3s
OS:               tumbleweed
public network:   10.20.190.0/24 
Proceed with deployment (y=yes, n=no, d=show details) ? [y]: y
=== Running shell command ===
vagrant up --no-destroy-on-error --provision
Bringing machine 'master' up with 'libvirt' provider...
Bringing machine 'node1' up with 'libvirt' provider...
Bringing machine 'node2' up with 'libvirt' provider...
Bringing machine 'node3' up with 'libvirt' provider...
Bringing machine 'node4' up with 'libvirt' provider...

[...
  wait a few minutes
  (there's lots more log information output here in real life)
...]

=== Deployment Finished ===
 You can login into the cluster with:
 $ sesdev ssh k3s

…and then I can do this:

> sesdev ssh k3s
Last login: Fri Mar 24 11:50:15 CET 2023 from 10.20.190.204 on ssh
Have a lot of fun…

master:~ # kubectl get nodes
NAME     STATUS   ROLES                  AGE     VERSION
master   Ready    control-plane,master   5m16s   v1.25.7+k3s1
node2    Ready                     2m17s   v1.25.7+k3s1
node1    Ready                     2m15s   v1.25.7+k3s1
node3    Ready                     2m16s   v1.25.7+k3s1
node4    Ready                     2m16s   v1.25.7+k3s1 

master:~ # kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-79f67d76f8-rpj4d   1/1     Running     0          5m9s
kube-system   metrics-server-5f9f776df5-rsqhb           1/1     Running     0          5m9s
kube-system   coredns-597584b69b-xh4p7                  1/1     Running     0          5m9s
kube-system   helm-install-traefik-crd-zz2ld            0/1     Completed   0          5m10s
kube-system   helm-install-traefik-ckdsr                0/1     Completed   1          5m10s
kube-system   svclb-traefik-952808e4-5txd7              2/2     Running     0          3m55s
kube-system   traefik-66c46d954f-pgnv8                  1/1     Running     0          3m55s
kube-system   svclb-traefik-952808e4-dkkp6              2/2     Running     0          2m25s
kube-system   svclb-traefik-952808e4-7wk6l              2/2     Running     0          2m13s
kube-system   svclb-traefik-952808e4-chmbx              2/2     Running     0          2m14s
kube-system   svclb-traefik-952808e4-k7hrw              2/2     Running     0          2m14s

…and then I can make a mess with kubectl apply, helm, etc.

One thing that sesdev knows how to do is deploy VMs with extra virtual disks. This functionality is there for Ceph deployments, but there’s no reason we can’t turn it on when deploying k3s:

> sesdev create k3s --num-disks=2
> sesdev ssh k3s
master:~ # for node in \
    $(kubectl get nodes -o 'jsonpath={.items[*].metadata.name}') ;
    do echo $node ; ssh $node cat /proc/partitions ; done
master
major minor  #blocks  name
 253        0   44040192 vda
 253        1       2048 vda1
 253        2      20480 vda2
 253        3   44016623 vda3
node3
major minor  #blocks  name
 253        0   44040192 vda
 253        1       2048 vda1
 253        2      20480 vda2
 253        3   44016623 vda3
 253       16    8388608 vdb
 253       32    8388608 vdc
node2
 major minor  #blocks  name
 253        0   44040192 vda
 253        1       2048 vda1
 253        2      20480 vda2
 253        3   44016623 vda3
 253       16    8388608 vdb
 253       32    8388608 vdc
node4
 major minor  #blocks  name
 253        0   44040192 vda
 253        1       2048 vda1
 253        2      20480 vda2
 253        3   44016623 vda3
 253       16    8388608 vdb
 253       32    8388608 vdc
node1
 major minor  #blocks  name
 253        0   44040192 vda
 253        1       2048 vda1
 253        2      20480 vda2
 253        3   44016623 vda3
 253       16    8388608 vdb
 253       32    8388608 vdc

As you can see this gives all the worker nodes an extra two 8GB virtual disks. I suspect this may make sesdev an interesting tool for testing other Kubernetes based storage systems such as Longhorn, but I haven’t tried that yet.

,

Michael StillMinor questions in Linux file semantics

I’ve known for a long time that if you delete a file on Unix / Linux but that file is open somewhere, the blocks used by the file aren’t freed until that user closes the file (or is terminated), but I was left wondering about some other edge cases.

Shaken Fist has a distributed blob store. It also has a cache of images that virtual machines are using. If the blob store and the image cache are on the same filesystem, sometimes the image cache entry can be a hard link to an entry in the blob store (for example, if the entry in the blob store doesn’t need to be transcoded before use by the virtual machine). However, if they are on different file systems, I instead use a symbolic link.

This raises questions — what happens if you rename a file which is open for writing in a program? What happens if you change a symbolic link to point somewhere else while it is open? I suspect in both cases the right thing happens, but I decided I should test these theories out.

First off, let’s cover the moving a file which is being written to case. Specifically, moving the file on the same filesystem. I wrote this little test program:

#!/usr/bin/python3

import datetime
import time

with open('a', 'w') as f:
    try:
        while True:
            f.write('%s\n' % datetime.datetime.now())
            time.sleep(1)

    except KeyboardInterrupt:
        f.close()

In one terminal I set it running. In another I then renamed ‘a’ to ‘b’ and waited a bit. The short answer? The newer writes from my script ended up in ‘b’ correctly. This makes sense when you remember that files don’t have names in most Unix filesystems — a directory has dirents with names, and they point to an inode. The open program is changing the content of an inode and associated blocks, and that’s quite separate from changing the dirent that points to that inode.

Secondly, what happens if I have a symlink to a different filesystem, move the file on that other filesystem and then update the symlink? All of course while the file is in use?

Unsurprisingly it works just like the previous example — the open file continues to be updated regardless of the move and the change of symlink.

This is good, because it makes re-sharding the blob store in Shaken Fist much easier. So there you go.

,

Michael StillMalware Analyst’s Cookbook and DVD

Another technical book, this time because my employer lets me buy random technical books as long as I pinky swear to read them and this one sounded interesting and got good reviews.

First off, the book is a bit dated given its from 2011 — there are lots of references to Ubuntu 10.10 for example and they say to avoid Python 3, which has its historical charm. This is unfortunate given the first section of the book talks about setting up honeypots to collect malware to examine, but Dionaea for example had its last commit in 2021. I am left wondering if there are more modern honey pot systems that people use these days.

Secondly the book is definitely a cookbook and that’s on me for not noticing this about the book before buying it — its a series of recipes / scripts that do interesting things with malware. That said, it isn’t really teaching a cohesive set of skills, its more of a series of stepping stones along the path you might follow. I think that’s an unintended piece of important learning — books with “cookbook” or “recipes” in their title probably aren’t very good as an overview of a topic area. My bad.

That said, some parts of the book are very good — the discussion of whois, DNS, and Real Time Black Lists (RTBLs) is helpful and less focussed on providing scripts you could run. The discussion of how to log changes to a Windows system, detect attempts to hide files in NTFS filesystems, and detect changes to registry hives were interesting in an abstract way, but perhaps obvious to someone who actually uses Windows.

Overall, I’m a bit disappointed in this book and it will be exhiled to a shelf at the office as a punishment.

Malware Analyst's Cookbook and DVD Book Cover Malware Analyst's Cookbook and DVD
Michael Ligh, Steven Adair, Blake Hartstein, Matthew Richard,
Computers
John Wiley & Sons
November 2, 2010
747

A computer forensics "how-to" for fighting malicious code and analyzing incidents. With our ever-increasing reliance on computers comes an ever-growing risk of malware. Security professionals will find plenty of solutions in this book to the problems posed by viruses, Trojan horses, worms, spyware, rootkits, adware, and other invasive software. Written by well-known malware experts, this guide reveals solutions to numerous problems and includes a DVD of custom programs and tools that illustrate the concepts, enhancing your skills. Security professionals face a constant battle against malicious software; this practical manual will improve your analytical capabilities and provide dozens of valuable and innovative solutions Covers classifying malware, packing and unpacking, dynamic malware analysis, decoding and decrypting, rootkit detection, memory forensics, open source malware research, and much more.

,

Michael StillThe BeyondCorp papers

Google’s BeyondCorp effort would probably be what we would now call Zero Trust, although I am surprised by how little name recognition BeyondCorp has when I talk to security people about Zero Trust. Perhaps there are subtle differences between the two, but if there are they aren’t obvious to me. I find myself reading the relevant Usenix papers for BeyondCorp, so I figure I’ll post a summary of what I got from each paper here.

The earliest of these papers are quite old now (2014), especially for something the rest of the industry is only starting to talk a lot about at the moment. I wonder if there is a viable business model in watching what papers megacorps like Google publish, and the implementing them as commercialized products before the rest of the market catches on?

Either way, here’s a summary of the various papers from the perspective of an interested bystander…

BeyondCorp: a new approach to enterprise security is an introductory paper that introduces the idea of what we would now call Zero Trust networks. That is, that the internal corporate network is not categorized as especially trusted, but instead serves as an access mechanism to services which define their own trust of an end user. This trust is enforced by access gateways, and derived from metrics such as how recently OS updates have been installed on the requesting device. This is a good introduction to the concept, especially given its age.

BeyondCorp: design to deployment at Google — unfortunately this paper was less useful I think. It is higher level than the first paper, and provides fewer actionable insights for someone thinking of implementing Zero Trust.

BeyondCorp: the access proxy describes the high level architecture of the access proxy, which is the frontend which takes requests from clients and authenticates / authorizes them before passing them onto the protected services. There aren’t a lot of surprises here, but it is a good overview of what you might encounter along the way (non-HTTP protocols requiring a client side helper for example).

Migrating to BeyondCorp: Maintaining Productivity While Improving Security is a discussion of the process of transitioning the Google network to the new zero trust access methodology while not breaking users’ ability to get things done. This was implemented by partitioning the problem space into smaller more tractable problems, and then transitioning clients to the new non-priviledged VLAN as these problems were solved. A key component of this was an enterprise wide rollout of 802.1x to ensure device identity was well understood. This paper is largely descriptive — while it might provide inspiration to other implementations, it does not provide a complete roadmap, largely because every organization’s legacy applications will differ.

That said, one interesting idea is that the network rules to control traffic were implemented in two places — in the network layer for the new VLAN, but also in an iptables implementation on client machines. This meant that it was easy to add clients in test mode (with the local implementation), but turn it off again if things didn’t work out. It also meant that they could add enforcement in locations where the new VLAN had not yet been deployed.

Another interesting idea is the provisioning of micro-VPNs for harder to convert applications such as those requiring non-HTTP access to network resources. This looks to my modern eyes as a lot like what tailscale does — exposing a single application via a micro-VPN accessed from the client routing table.

BeyondCorp: The User Experience details the gradual reduction in the demand for “traditional” VPN connectivity as users were moved to BeyondCorp, even as users initially expected a more traditional approach. It covers other user support scenarios as well, but most of them are quite Google-specific (for example their loaner laptop program).

BeyondCorp: Building a Healthy Fleet is the final paper in the series and discusses defining the threats your are mitigating by undertaking a Zero Trust approach to network security. In the case of BeyondCorp a large amount of the benefit is derived from enforcing regular updates on the user endpoint fleet, as well as controlling who can access what service based on their business needs.

,

Simon LyallAudiobooks – February 2023

The Rules of the Game: Jutland and British Naval Command by Andrew Gordon

A very detailed account of the battle of Jutland and British Navel Culture. So detailed I gave up trying to follow the Audiobook but would work better if printed. 3/5

Wings of War: The World War II Fighter Plane that Saved the Allies and the Believers Who Made It Fly by David Fairbank White

The History of the P-51 Mustang through 3 people. Designer Edgar Schmued, Tommy Hitchcock, the man who fought for its adoption, and Don Blakeslee, an ace who flew it. 3/5

The Tombs of Atuan by Ursula K. Le Guin,

The 2nd Earthsea book. A girl grows up as a high priestess until one day Sparrowhawk comes to rob her temple. 3/5

1491: New Revelations of the Americas Before Columbus by Charles C. Mann

Nominally a history of the people’s of the pre-Columbian Americas. Covers the population, age and sophistication of the civilizations based on recent discoveries. 4/5

Geniuses at War: Bletchley Park, Colossus, and the Dawn of the Digital Age by David A. Price

A short book on the Bletchley Park code-breaking efforts of WW2. A general overview concentrating on a few characters with much left under-covered. 4/5

An Unfinished Life: John F. Kennedy, 1917–1963 by Robert Dallek

Good account of life and Presidency although only single volume and the audiobook is further abridged. Well worth it as a first JFK biography. 4/5

Tomorrow’s People: The Future of Humanity in Ten Numbers by Paul Morland

A review of some demographic trends and what they tell us how the world will look in the future. 4/5

Countdown to Pearl Harbor: The Twelve Days to the Attack by Steve Twomey

A chronicle of why America was unprepared for the Japanese Attack on Pearl Harbor. Detailed but a nice and interesting read. 4/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

,

Linux AustraliaCouncil Meeting 1st March 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Neill Cox (Secretary)
  • Russell Stuart (Treasurer)
  • Sae Rae Germaine (Council)
  • Jonathan Woithe (Council)

Apologies 

  • Marcus Herstik (Council)

Not Present

Meeting opened at 20:02 AEDT by Joel Addison  and quorum was achieved.

Minutes taken by Neill Cox

 

2. Log of correspondence

  • 22 Feb Enquiry from Bryce Torcello re linux.conf.au – Wil has responded
  • 25 Feb Miles Goodhew re LCA2022 close out. Miles has delivered the LCA gear to Neill and has now completed his tasks for LCA2022. He wishes the EverythingOpen organisers well but won’t be able to attend the conference. 
  • 27 Feb Russell Keith-Magee re Update to PyCon AU 2023 budget. 
  • 27 Feb Michael Richardson re DrupalSouth Wellington – Budget and Onboarding
  • PO Box: 5x Westpac statements 3x WWCC for EverythingOpen Conference

 

3. Items for discussion

  • Revised PyconAU budget:
    Council has concerns about the lower capacity budgets having online ticket sales but no platform to provide streaming.
    Council will discuss this further with the PyConAU organisers.  
  • DrupalSouth doing their conference in Wellington triggered several support queries from me to Stripe as all our banking links to anz.co.nz had been deleted, and I couldn’t figure out how to recreate them.  Stripe’s response said it’s no longer possible for LA to deposit funds into a NZ bank.  Depositing NZ$ card payments into an Australia bank seems prohibitively expensive, so Stripe is no longer a viable way of handling credit card payments in NZ$.  No immediate action required now as DrupalSouth is accepting payments via Lil’ Regie, but that won’t be an option for Symposium should we hold Pycon / EO / LCA in NZ.  Other options appear to be: use a different payment processor, or only accept AUD$ payments for NZ conferences.
    Finding a suitable payment processor may be difficult. PayPal have anti money laundering triggers that may not work for conferences that are quiet most of the year and then suddenly busy. Local (Australian) payment options can be expensive. Square seems to have stringent authentication requirements which can bite organisations which have no fixed address. Investigation will be required.
  • DrupalSouth have completed their financial induction, and have given us a budget, list team members, and agreed to abide by LA Policies.  The next steps are:
    • Consider their proposal (conference web site), and in particular their budget.
    • Pass a motion accepting them as a subcommittee.

Motion by Russell Stuart: That we accept the Drupal South budget and approve them as an LA subcommittee.
Seconded: Sae Ra Germaine

Passed unanimously

4. Items for noting

  • DrupalSouth financial induction done.  We are now waiting on them to provide event and team details, as described in last of induction.
  • EverythingOpen will run a “friends of LA” session (similar to ghosts sessions for LCA). There is a budget item for ghosts that will cover this option.
  • EverythingOpen Budget
    Sponsorship is going well, but at this stage ticket sales are lower than expected. At this stage a loss is likely.
  • EverythingOpen Charity fund matching
    Motion by Sae Ra Germaine: That LA matches up to $5,000 fundraising for Everybody’s Home (EverythingOpen’s chosen charity)

Seconded: Wil Brown

Passed: Unanimously

5. Other business

  • Admin team – Steve Walsh

Election held and as far as Steve knows it was fine, waiting for information from Julien re the possibility of identifying votes in the election.

Renewed opensource.org.au as a hold on opensource.au. Not delegated to anywhere, would perhaps be a good idea to sort that out so that it actually serves content. The other contender is opensource.net.au which is now held by a Fairfax company. 

Steve will fix the secretary@linux.org.au address to forward to Neill (neill.cox.la@gmail.com). There are issues with spam filtering on the office bearer email addresses. We have to be very careful of false positives.

The post Council Meeting 1st March 2023 – Minutes appeared first on Linux Australia.

,

Robert CollinsRustup CI / test suite performance

Rustup (the community package manage for the Rust language) was starting to really suffer : CI times were up at ~ one hour.

We’ve made some strides in bringing this down.

Caching factory for test scenarios

The first thing, which achieved about a 30% reduction in test time was to stop recreating all the test context every time.

Rustup tests the download/installation/upgrade of distributions of Rust. To avoid downloading gigabytes in the test suite, the suite creates mocks of the published Rust artifacts. These mocks are GPG signed and compressed with multiple compression methods, both of which are quite heavyweight operations to perform – and not actually the interesting code under test to execute.

Previously, every test was entirely hermetic, and usually the server state was also unmodified.

There were two cases where the state was modified. One, a small number of tests testing error conditions such as GPG signature failures. And two, quite a number of tests that were testing temporal behaviour: for instance, install nightly at time A, then with a newer server state, perform a rustup update and check a new version is downloaded and installed.

We’re partway through this migration, but compare these two tests:

fn check_updates_some() {
    check_update_setup(&|config| {
        set_current_dist_date(config, "2015-01-01");
        config.expect_ok(&["rustup", "update", "stable"]);
        config.expect_ok(&["rustup", "update", "beta"]);
        config.expect_ok(&["rustup", "update", "nightly"]);
        set_current_dist_date(config, "2015-01-02");
        config.expect_stdout_ok(
            &["rustup", "check"],
            for_host!(
                r"stable-{0} - Update available : 1.0.0 (hash-stable-1.0.0) -> 1.1.0 (hash-stable-1.1.0)
beta-{0} - Update available : 1.1.0 (hash-beta-1.1.0) -> 1.2.0 (hash-beta-1.2.0)
nightly-{0} - Update available : 1.2.0 (hash-nightly-1) -> 1.3.0 (hash-nightly-2)
"
            ),
        );
    })
}
fn check_updates_some() {
    test(&|config| {
        config.with_scenario(Scenario::ArchivesV2_2015_01_01, &|config| {
            config.expect_ok(&["rustup", "toolchain", "add", "stable", "beta", "nightly"]);
        });
        config.with_scenario(Scenario::SimpleV2, &|config| {
        config.expect_stdout_ok(
            &["rustup", "check"],
            for_host!(
                r"stable-{0} - Update available : 1.0.0 (hash-stable-1.0.0) -> 1.1.0 (hash-stable-1.1.0)
beta-{0} - Update available : 1.1.0 (hash-beta-1.1.0) -> 1.2.0 (hash-beta-1.2.0)
nightly-{0} - Update available : 1.2.0 (hash-nightly-1) -> 1.3.0 (hash-nightly-2)
"
            ),
        );
            })
    })
}

The former version mutates the date with set_current_dist_date; the new version uses two scenarios, one for the earlier time, and one for the later time. This permits the server state to be constructed only once. On a per-test basis it can move as much as 50% of the time out of the test.

Single binary for the integration test suite

The next major gain was moving from having 14 separate integration test binaries to just one. This reduces the link cost of linking the test binaries, all of which link in the same library. It also permits us to see unused functions in our test support library, which helps with cleaning up cruft rather than having it accumulate.

Hard linking rather than copying ‘rustup-init’

Part of the test suite for each test is setting up an installed rustup environment. Why not start from scratch every time? Well, we obviously have tests that do that, but most tests are focused on steps beyond the new-user case. Setting up an installed rustup environment has a few steps, but particular ones are copying a binary of rustup into the test sandbox, and hard linking it under various names: cargo, rustc, rustup etc.

A debug build of rustup is ~20MB. Running 400 tests means about 8GB of IO; on some platforms most of that IO won’t hit disk, on others it will.

In review now is a PR that changes the initial copy to a hardlink: we hardlink the rustup-init built by cargo into each test, and then hardlink that to the various binaries. That saves 8GB of IO, which isn’t much from some perspectives, but it adds pressure on the page cache, and is wasted work. One wrinkle is a very low max-links limit on NTFS of 1023; to mitigate that we count the links made to rustup-init and generate a new inode for the original to avoid failures happening.

Future work

In GitHub actions this lowers our test time to 19m for Linux, 24m for Windows, which is a lot better but not great.

I plan on experimenting with separate actions for building release artifacts and doing CI tests – at the moment we have the same action do both, but they don’t share artifacts in the cache in any meaningful way, so we can probably gain parallelism there, as well as turning off release builds entirely for CI.

We should finish the cached test context work and use it everywhere.

Also we’re looking at having less integration tests and more narrow close to the code tests.

Michael StillCisco CyberOps Associate: Official Cert Guide

I don’t think I’ve really reviewed a technical book here before, but I read the thing so I guess I should. This book is the certification guide for a “Cisco CyberOps Associate” certification, which is what they now call the CCNA Security qualification. Its a relatively junior certification, qualifying you to be a level one operator in a Security Operations Centre (SOC).

I read this book because I took a Cisco NetAcad course for the associated certification in the second half of 2022 (although it has continued to be a thing I plug away at in 2023). That was mainly motivated by a desire to more about a field that is clearly important, but hasn’t been core to my personal career.

This book is reasonably well written and readable — I’d read a chapter in the evening after work and its wasn’t a huge chore to churn though. I certainly learned things along the way, even if the certification seems to suffer from a desire to have everyone rote learn a lot of acronyms, which seems like a common ailment in the industry (AWS Certified Cloud Practitioner, I’m looking at you).

My main critism is of the qualification itself, which is that it is quite Cisco centric — almost all examples of the implementation of a technology are a Cisco product, which is great if you’re trying to demonstrate the depth of Cisco’s portfolio, but isn’t great if you’re competing with less vendor centric certification options. This is in contrast to the CCNA content, which feels more vendor neutral to me because its more fundamental.

That said, this book wasn’t a waste of my time and I learned stuff — which I guess is mission accomplished for a technical book?

Cisco Cyberops Associate Cbrops 200-201 Official Cert Guide Book Cover Cisco Cyberops Associate Cbrops 200-201 Official Cert Guide
Omar Santos
Computers
Cisco Press
August 6, 2020
900

Modern organizations rely on Security Operations Center (SOC) teams to vigilantly watch security systems, rapidly detect breaches, and respond quickly and effectively. To succeed, SOCs desperately need more qualified cybersecurity professionals. Cisco's new Cisco Certified CyberOps Associate certification prepares candidates to begin a career working as associate-level cybersecurity analysts within SOCs.

,

Lev Lafayette2022 HPC Training Utilisation and Results

Unique identifiers for 263 users who received HPC training in 2022 was determined from collected attendee records. Note that users may enrol in multiple courses (e.g., Introduction to Spartan, Advanced Spartan, Parallel Processing, etc) and may return for revision. All these users are counted once only.

From the unique users a total of 212 usernames could be determined from email addresses. When enrolling for training users do not include their Spartan usernmae or their university ID; sometimes they don't even use a university email address, despite requests.

There were 97 users who established an account but did not use Spartan (compute hours = 0). Of the remaining 115 users the total of job hours was determined from trained users was 6280454, after they received training. This calculation ensured that users who had already run jobs on Spartan prior to receiving training was not counted. e.g.,

$ sreport cluster AccountUtilizationByUser cluster=spartan user=$username start=2022-11-01 end=2022-12-31 -t hours

The total allocated hours of cluster utilisation 11597951, from the command:

$ sreport cluster Utilization cluster=spartan start=2022-11-01 end=2022-12-31 -t hours

The means that at least 54.14% of cluster utilisation in 2022 was conducted by users after receiving training.

The following steps are recommended to improve record-keeping and utilisation.

1) Emphasising the need to enrollees to use University of Melbourne email addresses only, and rejecting applications that do not do this.

2) Contacting those who attended training but did not use Spartan to ascertain why this was the case.

,

Michael StillExploring more efficient remote large file storage

My primary personal project is a thing called Shaken Fist these days — it is an infrastructure as a service cloud akin to OpenStack Compute, but smaller and simpler. Shaken Fist doesn’t have an equivalent to the OpenStack Image service, instead letting your describe your instance images by a standard URL. One of the things Shaken Fist does to be easier to use is it maintains an official repository of common images, which allows users to refer to those images with a shorthand syntax instead of a complete URL. The images also contain small customizations (mainly including the Shaken Fist in-guest agent), which means I can’t just use the official upstream cloud images like OpenStack does.

The images were stored at DreamHost until this week, when a robot decided that they looked like offline backups, despite being served to the Internet via HTTP and being used regularly (although admittedly not frequently). DreamHost unilaterally decided to delete the web site, so now I am looking for new image hosting services, and thinking about better ways to build an image store.

(Oh, and recommending to anyone who asks that they consider using someone less capricious than DreamHost for their hosting needs).

All of this got me thinking. What would be the requirements for a next-generation image store for Shaken Fist? Given Shaken Fist is a personal project with not a lot of delivery pressure, I’ve been trying for the last year or two to “take the tangent” when one appears. In general I try to develop those ancillary things as separate sub-projects, in the hope that they’ll be useful to other people one day. Examples of tangents include Occystrap (OCI image support in python), and Clingwrap (a tool to build “field service dumps” of machine state to aid in debugging).

What would a better image store look like if I took that tangent? Here’s what I came up with:

  • Efficient storage of images where the changes between the images are small: I do daily builds for the images, and theorize that therefore the images should be fairly similar day to day. It would be nice to only store the delta somehow when adding a new image.
  • Improved cachability for clients: if they possess an older image which overlaps with a newer one, they should only have to download the delta.
  • Support for cloud native object stores: there are a variety of inexpensive cloud object stores these days, and it would be nice to harness one of those for storage. The big limitation here is that in general these object stores do now allow for directory listing, so they’re good for key / value lookups where you already know the key, but not good for iterating all keys.

A specific non-goal is being fast to encode into the new format — that only happens once on a single build machine, and I am willing to accept some additional computation in return for reduced storage costs and faster downloads for my users.

My initial naive implementation uses the following algorithm:

  • Split the file into chunks and calculate a sha512 checksum of that chunk.
  • If the remote store already has a chunk with that checksum, do not upload.
  • If it does not, then compress the chunk using gzip and upload it.
  • Add the sha512 checksum to the list of chunks the file is composed of.
  • After the entire file has been processed, emit a file containing metadata about the file, including the list of chunks.

My thinking is that the metadata file is the one you’d refer to when downloading an image, and the chunks would then be fetched as a second stage. I note that this approach has similarities to how Docker layers are stored in container image repositories.

There are some obvious experiments possible here. For example, what is the optimal chunk size? I should note that the chunk size can’t be too small, because each unique chunk becomes an object, and for some object stores (such as Linux filesystems), there are limits on the number of objects we can reasonably store.

I therefore took my 113gb collection of CentOS 8 images, and tried a variety of chunk sizes. I know I said above that I don’t particularly care about processing time, but I do think its good to keep track of. Therefore, I also measured the “no-write-IO” time to process the repository — that is I encoded the respository and then encoded it again to the same destination. This gives me a measure of processing time while not including any writes to the destination which hopefully minimises the noise from a busy spinning rust disk array as much as possible. Here are the numbers:

Chunk size (mb)ChunksRepository size (gb)Processing time (minutes, no writes)Respository size (%age of original size)
196,699911781%
250,824961385%
334,780981187%
426,5031001688%
521,4841011189%

Or graphically:

A graph showing preformance of BlockStash on compressed images with varying chunk sizes.
BlockStash performance on compressed images

But wait! The default in my previous image store used compressed qcow2 images — that is they have been compressed with the DEFLATE algorithm. These images are still sparse regardless of the compression used, so we are unlikely to see large blocks of zeros in the data.

The compression in the source images means we’re trying to chunk up files with a streaming compression algorithm that likes to use references to previous data seen earlier in the file. It therefore seems likely that performance will change with uncompressed input images. Here’s the same experiment, but with 241gb of uncompressed images passed (the same images as before, just decompressed):

Chunk size (mb)ChunksRepository size (gb)Processing time (minutes, no writes)Respository size (%age of original size)
1175205724264%
299458823473%
370520873177%
454494902980%
544835922881%

Or graphically:

BlockStash performance on uncompressed images

As an aside, there’s a notable flaw in this first chunking approach, because a change early in a file which offsets the chunks will result in each chunk being new. I haven’t thought of a computationally reasonable fix for that, so such is life for now. In theory, it could be fixed in this proposed format by inserting a small “shim” chunk which has that early change, but computing all possible hashes for all possible sliding blocks sounds super expensive to me. Rabin-Karp rolling hashes look promising for helping with this issue, but I haven’t persued it further because my target data is disk images which are block editted, not insertion editted.

So what option should I pick? My reading of the data is that there are sigificant server side storage wins (and an associated reduction in the need for client downloads) if I go for a smaller chunk size starting with uncompressed data, but I have concerns about the number of chunks a file download might incur — is downloading 1,024 1mb chunks actually faster than downloading a single 1gb file? I think that’s perhaps a matter for another post, as I’ll need to do some more benchmarking there.

,

Linux AustraliaCouncil Meeting 15th February 2023 – Minutes

1. Meeting overview and key information

Present

  • Joel Addison (President)
  • Wil Brown (Vice-President)
  • Neill Cox (Secretary)
  • Russell Stuart (Treasurer)
  • Sae Ra Germaine  (Council)
  • Marcus Herstik (Council)
  • Jonathan Woithe (Council)

 

Apologies 

  • None

 

Not Present

  • None

 

Meeting opened at 20:08 AEDT by Joel Addison  and quorum was achieved.

Minutes taken by Neill Cox

 

2. Log of correspondence

  • Email from Dwight Walker re CiviCRM install notes. Joel and Steve Walsh have already responded. Further emails from Dwight received Feb 13. Steve has been in Fiji for work for the last two weeks. Dwight is attempting to setup a development environment rather than waiting any longer.
  • Email from Richard Jones “Kicking off the PyCon AU 2023 financials”. Joel has responded. 
  • Two emails from YouTube re “New YouTube Partner Program terms coming this week”. Joel has actioned(TBC)
  • Email from Russell Stuart re “NSW Fair trading annual return”. Russel has actioned this. Money has been paid.
  • Email from Zoom re change to account billing information (Russell). New credit card has been recorded on the account.
  • Email thread re Linux Australia Activity Statement Oct..Dec 2022 (Russell and Sonny Bonang (Activity Statement lodged and accountant’s invoice paid)
  • Email from Kathy Reid re Nat Torkington’s son William who sadly took his own life. Wil has responded.
  • Email from Miles Goodhew re “Moving the LCA stuff to Brisbane”. Miles has a quote for $1,145 (from Dawson Removals), insurance would be $35/$1,000 of value. Miles has two other quotes from Kent Moving ($745 plus insurance) and Grace Removals for $1,228.57. Miles would like his shed back.
  • Email from Kathy Reid re “Seeking to allocate $AUD 1000 from budgeted LA funds to Media & Communications Subctte for rewards and recognition”
  • PO Box payment due soon
  • Richard Jones has requested an update on PyConAU 2023. Joel has responded, basically saying we will reply as soon as possible after the first meeting of the new council.
  • Betsy Waliszewski  re OSI’s 25th Anniversary and Everything Open
  • Steve Walsh re APNIC Membership Renewal. Renewal invoice has been paid.
  • Russell Stuart, Michael Richardson & Owen Lansbury re Xero and Westpac Setup for Drupal South
  • PO Box relocated to Kent Street
  • Westpac and ANZ bank statements
  • Volunteer WWCC notice received
  • secretary@ emails still going to Clinton

3. Items for discussion

  • Shipping LCA stuff
    • Much of the LCA gear is now probably of limited value. Marcus has offered to transport from Canberra if it will fit.
    • We have pictures but they lack scale.
    • Steve Walsh may be able to ship things via AARNet.
    • Joel will look through the list of gear from Miles to decide what to keep and what we can get rid of.
    • Russel suggests that we make it Everything Open’s problem.
    • Sae Ra points out that there is nowhere to put it in Melbourne.
    • Action Item: Joel to go through the list to see what can be culled. Then we’ll decide how to move the remainder..
  • Media and Communications Subcommittee funding request
    • We currently allow the admin team to manage their own budget, as do the various event subcommittees for events.
    • $2,000 is already budgeted for Rewards and Recognition and $1,000 for Advertising and Marketing, so effectively this is us giving permission to Kathy to spend money that she has already been authorised to use.
    • Motion by Joel: That we allocate $1,000 to the media subcommittee from the media budget. Carried unanimously
  • PO Box Payment due soon
    • Email was sent to Neill who has now forwarded it to Council and Treasurer for payment.
    • Joel will ask Steve to update the secretary@ address to go to Neill
  • OSI 25th Anniversary
    • OSI would like to celebrate this at EO2023. Possibly cupcakes/afternoon tea. This will be handled by the EO2023 organising committee.
  • APNIC Renewal
    • Has been paid.
  • Drupal South
    • Drupal South will be held in New Zealand this year, dates to be confirmed.  Proposal is yet to be formally accepted by the LA council. Most recent correspondence was that they were still finalising their budget. Looks to be very similar to last year’s conference, including the use of an external conference organiser.
  • Conflict of interest register.
    • Joel will share a link and we should each fill out as appropriate
  • William Torkington
    • Wil to add an in memorandum page to the website and include William. Sae Ra to send a list of other people to be included.
  • Working With Children
    • These are needed for organisers of EO2023, but also probably LA council members.
    • Takes some time to process. Registered mail seems to slow the process down. Unfortunately
    • sending 100+ points of ID through the mail unregistered is not something to make people happy.
  • PyCon AU 2023 – Budget 2023 – ACC
    • PyConAU were able to use a deposit paid for the venue when it was originally planned to be held in Adelaide pre-covid.
    • The attendance numbers do not seem to reflect the current economic climate. However, given that the choice is between losing $1,000 at the bare bones level or $100,000 if we forfeit the deposit the best option is to go ahead with the conference.
    • Joel will respond to Richard giving approval for the conference to be announced, but asking that the budget be revisited with expectations of fewer attendees.

 

4. Items for noting

  • Russell is holding a financial induction for Drupal South.  Drupal can make it on  UTC 20:00..23:00 on Feb 20, 21, or 22.  Do any committee members want to join, and if so lets settle on a time.
    • Russell will finalise a time and send zoom invites.

5. Other business

  • None

The post Council Meeting 15th February 2023 – Minutes appeared first on Linux Australia.

,

Michael StillThis is going to hurt

This book is lots of things: honest, funny, and ultimately heart breaking. I don’t remember how I came across it, but its a good read for when travelling as the diary format means you can put it down whenever you need to do something else.

I’m left wondering how the Australian medical system compares to the NHS — I know we have more patient choice and flexibility — but I wonder what its like for those working within the system.

Either way I definitely recommend this book.

This is Going to Hurt Book Cover This is Going to Hurt
Adam Kay
Biography & Autobiography
Pan Macmillan
2018
279

As soon as Adam Kay set foot on a hospital ward for the first time, he realised there's quite a lot they don't teach you at medical school ... His diaries from the NHS front line - scribbled in secret after long nights, endless days and missed weekends - are hilarious, horrifying and heartbreaking by turns. This Is Going to Hurt is everything you wanted to know about being a junior doctor, and more than a few things you really didn't. And yes, it may leave a scar.

,

Simon LyallAudioBooks – January 2023

Colditz Prisoners of the Castle by Ben Macintyre

A good contrast to the “Boys Own” versions by Pat Reid I read as a kid. Covers lots of other viewpoints including from the Germans. Recommended 4/5

Project Hail Mary by Andy Weir

Last read July 2021. A semi-repeat of The Martin where a lone astronaut has to science the shit out of a bad situation. This time to save humanity. 4/5

Seven Games: A Human History by Oliver Roeder

Working through increased complexity of Checkers, backgammon, chess, and Go. Poker, Scrabble, and bridge the author looks at how humans and computers play them. 4/5

Daily Rituals: How Artists Work by Mason Curry

161 short articles about the work habits of authors, artists, composers and the like. Interesting with some ideas one can potentially adopt. 3/5

The Extraordinary Life of an Ordinary Man: A Memoir by Paul Newman

Based on tapes recordings made by the actor and those that knew him. Honest and Deep rather than broad and concentrating on his early life and career. 4/5

A Wizard of Earthsea by Ursula K. Le Guin

Classic Children’s Fantasy story that I haven’t read since I was a kid. Told in a very epic tone and language. Good although I missed the map on audiobook. 4/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

,

Tim SerongHack Week 22: An Art Project

Back in 2012, I received a box of eight hundred openSUSE 12.1 promo DVDs, which I then set out to distribute to local Linux users’ groups, tech conferences, other SUSE crew in Australia, and so forth. I didn’t manage to shift all 800 DVDs at the time, and I recently rediscovered the remaining three hundred and eighty four while installing some new shelves. As openSUSE 12.1 went end of life in May 2013, it seemed likely the DVDs were now useless, but I couldn’t bring myself to toss them in landfill. Instead, given last week was Hack Week, I decided to use them for an art project. Here’s the end result:

Geeko mosaic made of cut up openSUSE DVDs, on a 900mm x 600mm piece of plywood

Making that mosaic was extremely fiddly. It’s possibly the most annoying Hack Week project I’ve ever done, but I’m very happy with the outcome 🙂

The backing is a piece of 900mm x 600mm x 6mm plywood, primed with some leftover kitchen and bathroom undercoat, then spray pained black. I’d forgotten how bad spray paint smells, but it makes for a nice finish. To get the Geeko shape, I took the official openSUSE logo, then turned it into an outline in Inkscape, saved that as a PNG, opened it in GIMP, and cut it into nine 300mm x 200mm pieces which I then printed on A4 paper, stuck together with tape, and cut out to make a stencil. Of course, the first time I did that, nothing quite lined up, so I had to reprint it but with “Ignore page margins” turned off and “Draw crop marks” turned on, then cut the pages down along the crop marks before sticking them together the second time. Then I placed the stencil on the backing, glued the eye down (that just had to be made from the centre of a DVD!) and started laying out cut up DVD shards.

Geeko mosaic work in progress

I initially tried cutting the DVDs with tin snips, which is easy on the hands, but had a tendency to sometimes warp the DVD pieces and/or cause them to delaminate, so I reverted to a large pair of scissors which was more effort but ultimately less problematic.

After placing the pieces that made up the head, tail, feet and spine, and deciding I was happy with how they looked, I glued each piece down with superglue. Think: carefully pick up DVD shard without moving too many other shards, turn over, dab on a few tiny globs of superglue, lower into place, press for a few seconds, move to next piece. Do not get any superglue on your fingers, or you’ll risk sticking your fingers together and/or make a gluey mess on the shiny visible side of the DVD shards.

It was another three sessions of layout-then-glue-down to fill in the body. I think I stuck my fingers together about six, or eight, or maybe twenty times. Also, despite my best efforts to get superglue absolutely nowhere near the stencil at all, when I removed the stencil, it had stuck to the backing in several places. I managed to scrape/cut that off with a combination of fingernails, tweezers, and the very sharp knife in my SLE 12 commemorative Leatherman tool, then touched up the remaining white bits with a fine point black Sharpie.

SLE 12 commemorative Leatherman tool (it seemed appropriate to use this)

Judging from the leftover DVD centre pieces, this mosaic used about 12 DVDs in all, which isn’t very many considering my initial stash. I had a few other ideas for the remainder, mostly involving hanging them up somehow, which I messed around with earlier on while waiting for the paint to dry on the plywood.

One (failed) idea was to use a cutting wheel on my Dremel tool to slice half way through a few DVDs, then slot them into each other to make a hanging thingy that would spin in the wind. I was unable to make a smooth/straight enough cut for this to work, and superglue doesn’t bridge gaps. You can maybe get an idea of what I was aiming at from this photo:

Four DVDs slotted into each other vertically, kinda, one with nasty superglue smear

My wife had an idea for a better way to do this, which is to take a piece of dowel, cut slots in the sides, and glue DVD halves into the slots using Araldite (that’s an epoxy resin, in case you didn’t grow up with that brand name). I didn’t get around to trying this, but I reckon she’s onto something. Next time I’m at the hardware store, I’ll try to remember to pick up some suitably sized dowel.

I did make one somewhat simpler hanging thingy, which I call “Geeko’s Tail (Uncurled)”. It’s just DVDs superglued together on the flat, hanging from fishing line, but I think it’s kinda cool:

No, it’s not an upside down question mark, it’s “Geeko’s Tail (Uncurled)”

Also, I’ve discovered that Officeworks has an e-waste recycling program, so any DVDs I don’t use in future projects needn’t go to landfill.

Update 2023-02-20: For photos of the mosaic, plus wallpapers made from the photos, see https://github.com/tserong/hackweek22

,

Lev LafayetteThe Importance of Supercomputing

Most people use their computers (which includes mobile phones) for communication, social media, games, entertainment, office applications, and the like. Most of the time these activities are not particularly onerous in terms of computing as such or do not lead to enormous benefits in productivity, inventions, and discovery. There is one field, however, rarely discussed, that does do this - and that is supercomputing. It is through supercomputing that we are witnessing the most important technological advances of our day, including astronomy, weather and climate forecasting, materials science and engineering, molecular modeling, genomics, neurology, geoscience, and finance - all with numerous success stories.

Usually, I draw a distinction between supercomputing and high-performance computing. Specifically, a supercomputer is any computer system that has exceptional computational power at a particular point in time, many (but not all) of which are measured in the bi-annual Top500 list. Once upon a time dominated by monolithic mainframes supercomputers, in a contemporary sense, are a subset of high-performance computing, which is typically arranged as a cluster of commodity-grade servers with a high-speed interconnect and message-passing software that allows the entire unit to be treated as a whole. One can even put together a "supercomputer" from Raspberry Pi systems, as the University of Southhampton illustrates.

How important is this? For many years now we've known that there is a strong association between research output and access to such systems. Macroeconomic analysis shows that for every dollar invested in supercomputing, there is a return of forty-four dollars in profits or cost-savings. Both these metrics are almost certainly going to increase in time; datasets and problem complexity are growing at a rate greater than the computational performance of personal systems. More researchers need access to supercomputers.

However, researchers do require training to use such systems. The environment, the interface, the use of schedulers on a shared system, the location of data, is all something that needs to be learned. This is a big part of my life; in the last week, I spent three days teaching researchers from the basic of using a supercomputer system to scripting jobs, to using Australia's most powerful system Gadi at NCI, along with contributions at a board meeting of the international HPC Certification Forum. It is often a challenging vocation, but I feel confident that it is making a real difference to our shared lives. For that, I am very grateful.

,

Colin CharlesLong Malaysians, Short Malaysia

I have long said “Long Malaysians, Short Malaysia” in conversation to many. Maybe it took me a while to tweet it, but this was the first example: Dec 29, 2021. I’ve tweeted it a lot more since.

Malaysia has a 10th Prime Minister, but in general, it is a very precarious partnership. Consider it, same shit, different day?

I just have to get off the Malaysian news diet. Malaysians elsewhere, are generally very successful. Malaysians suffering by their daily doldrums, well, they just need to wake up, see the light, and succeed.

In the end, as much as people paraphrase, ask not what the country can do for you, legitimately, this is your life, and you should be taking good care of yourself and your loved ones. You succeed, despite of. Politics and the state happens, regardless of.

Me, personally? Ideas are abound for how to get Malaysians who see the light, to succeed elsewhere. And if I read, and get angry at something (tweet rage?), I’m going to pop RM50 into an investment account, which should help me get off this poor habit. I’ll probably also just cut subscriptions to Malaysian news things… Less exposure, is actually better for you. I can’t believe that it has taken me this long to realise this.

Time to build.

The post Long Malaysians, Short Malaysia first appeared on Colin Charles Agenda.

Colin CharlesHello 2023

I did poorly blogging last year. Oops. I think to myself when I read, This Thing Still On?, I really have to do better in 2023. Maybe the catalyst is the fact that Twitter is becoming a shit show. I doubt people will leave the platform in droves, per se, but I think we are coming back to the need for decentralised blogs again.

I have 477 days to becoming 40. I ditched the Hobonich Techo sometime in 2022, and just focused on the Field Notes, and this year, I’ve got a Monocle x Leuchtturm1917 + Field Notes combo (though it seems my subscription lapsed Winter 2022, I should really burn down the existing collection, and resubscribe).

2022 was pretty amazing. Lots of work. Lots of fun. 256 days on the road (what a number), 339,551km travelled, 49 cities, 20 countries.

The getting back into doing, and not being afraid of experimenting in public is what 2023 is all about. The Year of The Rabbit is upon us tomorrow, hence why I don’t mind a little later Hello 2023 :)

Get back into the habit of doing. And publishing by learning and doing. No fear. Not that I wasn’t doing, but its time to be prolific with what’s been going on.

I better remember that.

The post Hello 2023 first appeared on Colin Charles Agenda.

,

Lev LafayetteInstalling VASP 6.x on x86_64 RHEL 7.9 Linux

In the past I have posted two sets of instructions for installing VASP (Vienna Ab-initio Simulation Package for quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set), each for VASP 5.X on an Opteron system. Now, many years later, I find myself in the position of having to install VASP once again.

The installation approach is still pretty horrible but it has improved a great deal. Previously there was a small mountain of makefiles for different architectures and one had to find a file that was "close enough" and modify as required. This process is still required, but the quantity of makefiles is dramatically reduced with improved abstraction, directory management, and a test suite.

The structure (once extracted) is as follows:

                   vasp.X.X.X (root directory)
                                |
          ------------------------------------------------
         |        |        |         |          |         |
        arch     bin     build      src     testsuite   tools

* `root/`. Holds the high-level makefile and several subdirectories.
* `root/src`. Holds the source files of VASP and a low-level makefile.
* `root/arch`. Holds a collection of `makefile.include.*` files.
* `root/build`. The different versions of VASP, i.e., the standard, gamma-only, non-collinear, and CUDA-GPU versions will be build in separate subdirectories of this directory.
* `root/bin`. Here make will store the binaries.
* `root/testsuite`. Holds a suite of correctness tests to check your build.
* `root/tools`. Holds several python scripts related to the (optional) use of HDF5 input/output files.

Installation involves copying one the makefile.include.xxx files to the root directory as makefile.include, modifying it running make. The most straightforward, used in this example, is makefile.include.linux_gnu.

Within the makefile one will have to enter the values of the Fortran library directory, the LIBDIR for Blas, LAPACK, SCALAPACK, and FFTW. c.f.,

# LIBDIR     = /opt/gfortran/libs/
LIBDIR     = /usr/local/easybuild-2019/easybuild/software/core/gcccore/10.2.0/lib64/lib
BLAS       = -L$(LIBDIR) -lrefblas
LAPACK     = -L$(LIBDIR) -ltmglib -llapack
BLACS      = 
SCALAPACK  = -L$(LIBDIR) -lscalapack $(BLACS)

LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)

In this particular installation, there is the use of the EasyBuild foss/2020b toolchain, which consists of GCC/10.2.0 and OpenMPI 4.0.5. Once that toolchain is loaded then one can also load FFTW/3.3.8, scalapack/2.1.0, openblas/0.3.12. Note that loading the modules will not be read by the VASP makefile. They still have to be hard-coded - it is convenient however when checking the PATH to the libraries.

The above code snippet from the makefile is a little deceptive. Something like the following is recommended instead:

# LIBDIR     = /opt/gfortran/libs/
LIBDIR     = /usr/local/easybuild-2019/easybuild/software/core/gcccore/10.2.0/lib64/lib
# BLAS       = -L$(LIBDIR) -lrefblas
# LAPACK     = -L$(LIBDIR) -ltmglib -llapack
BLACS      = 
# SCALAPACK  = -L$(LIBDIR) -lscalapack $(BLACS)

OPENBLAS_ROOT ?= /usr/local/easybuild-2019/easybuild/software/compiler/gcc/10.2.0/openblas/0.3.12/
BLASPACK    = -L$(OPENBLAS_ROOT)/lib -lopenblas

SCALAPACK_ROOT ?= /usr/local/easybuild-2019/easybuild/software/mpi/gcc/10.2.0/openmpi/4.0.5/scalapack/2.1.0
SCALAPACK   = -L$(SCALAPACK_ROOT)/lib -lscalapack

LLIBS      += $(SCALAPACK) $(BLASPACK)

# FFTW       ?= /opt/gfortran/fftw-3.3.6-GCC-5.4.1
FFTW       ?= /usr/local/easybuild-2019/easybuild/software/mpi/gcc/10.2.0/openmpi/4.0.5/fftw/3.3.8
LLIBS      += -L$(FFTW)/lib -lfftw3 -lfftw3_omp
INCS       = -I$(FFTW)/include

There is a further issue. If one is using GCC 10.x or greater there will be an argument mismatch. An error will occur like the following:

Error: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)

To get around this an additional Fortran flag must be added, resulting in:

#  For gcc-10 and higher require -fallow
FFLAGS     = -w -march=native -fallow-argument-mismatch

Another interesting error is that VASP has only been built for up to GCC7. The use of GCC 10 and MPI will result in an error and the reader_base.F file needs to be patched. This has been discussed on the VASP forums, which also has a copy of the patchfile. Modify the headers if necessary and apply the patch. e.g.,

patch < reader.patch 
patching file reader_base.F

Following this, an incremental build of the three core VASP binaries should work.

make std
make gam
make ncl

,

Simon LyallAudiobooks – December 2022

The Years of Lyndon Johnson. Book Four: The Passage of Power by Robert Caro

Covers 1958-1964. Especially the 1960 Democratic primary and election, Johnson’s unhappy Vice Presidency and the first months of his Presidency. As good as the others in the series. 4/5

England’s Villages: An Extraordinary Journey Through Time by Dr Ben Robinson

An archaeologist writes about the evolution of English Villages, their people, buildings, names and forms. Okay but not exceptional. 3/5

Jefferson: Architect of American Liberty by John B. Boles

A good single volume biography. Works hard to explain Jefferson’s attitudes especially on slavery. Good coverage and easy to follow. 4/5

Shutdown: How Covid Shook the World’s Economy by Adam Tooze

Covering roughly 2020 plus a few months on each side it mostly concentrates on the government and central bank measures to stabilise economies. 3/5

Doing Good Better: Effective Altruism and How You Can Make a Difference by William MacAskill

A Introduction to Effective Ultruism and how you can do the most good in the world via carefully picking charities to give to and other alternatives. 4/5

Leave the Gun, Take the Cannoli: The Epic Story of the Making of The Godfather by Mark Seal

Covers the writing of the book by Puzo, adapting and then filming it. Lots of Behind the scenes stories. A fun read 4/5

The 2020 Commission Report on the North Korean Nuclear Attacks Against the United States by Jeffrey Lewis

A future/alternative history where Trump’s America fights North Korea. Well done and relatively plausible. 4/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

,

Simon LyallDonations 2022

Each year I do the majority of my Charity donations in early December (just after my birthday) spread over a few days (so as not to get my credit card suspended).

I do a blog post about it to hopefully inspire others. See previous years: 2021, 2020, 2019, 2018, 2017, 2016, 2015

All amounts are in $US unless otherwise stated

General Charities

$750 to Givewell Top Charities fund . This was previously called their “Maximum impact fund”.

Software and Internet Infrastructure Projects

Last year I donated $100 each to SPI and SFC but this year I dropped it to $50 each and did direct donations to Python and Syncthing. I’m not sure which is the best strategy.

Others including content creators

Payments via Patreon

Current as of mid-December 2022

  • $2/month to Daniel King to make Chess videos
  • $1/month to Chris Stuckmann who does movie reviews
  • $2/month to The Prancing Pony Podcast who make a podcasts show about J R R Tolkien
  • $1/month to Joe Snodow who runs funny twitter accounts.
  • $1/month to Zach Weinersmith who creates SMBC Comic and other stuff
  • $1/video to The Nerdwriter who does Youtube videos
  • $1/month to CGP Grey who does Youtube Videos
  • $1/month to City Beautiful who is creating videos about cities and city planning.
  • $1/month to Alt Shift X who creates youtube videos
  • $2/month to Rose Eveleth who creates the Flash Forward podcast.
  • $1/month to RMTransit who does a Youtube channel on Transit.
  • $1/month to Quinn’s Ideas which is a Youtube Channel about Science Fiction (especially Dune)
  • $1/ month to Asianometry who creates youtube videos, mainly on Economics and the semiconductor industry.
  • $1/month to CityNerd who Videos on Cities and Transportation

Share

,

Simon LyallAudiobooks – November 2022

Calculating God by Robert J. Sawyer

Aliens arrive on present-day Earth and one befriends a Canadian paleontologist. These of religion & alien civilizations are covered. Good read. 3/5

Working: Researching, Interviewing, Writing by Robert A Caro

A series of articles on the author’s process & experiences researching and writing his biographies. Short but interesting. 4/5

Consider Phlebas by Iain M. Banks

A “spy story” within a interstellar conflict, it introduces “The Culture” civilization. Reasonable main character and lots of stuff for Hard Core SF fans. 3/5

The Hunt for Vulcan: . . . And How Albert Einstein Destroyed a Planet, Discovered Relativity, and Deciphered the Universe by Thomas Levenson

Fun story following a few main characters (Thomas Edison has a cameo). 4/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

,

Michael StillAnsible 7.0 onwards requires blocking IO from stdin, stdout, and stderr

Shaken Fist CI started failing this afternoon with this message logged:

ERROR: Ansible requires blocking IO on stdin/stdout/stderr.
Non-blocking file handles detected: <stdout>

Specifically this was happening when using ansible-galaxy to install some requirements, but the check is a more generic check than that was implemented by this ansible pull request, which appears to have been released with ansible-core 2.14 on November 8. That sat around until today, when ansible 7.0.0 was released and broke CI for me.

To be completely honest I’m not sure what’s happening here — somewhere in GitHub actions calling a shell script that calls ansible-galaxy the stdout file descriptor gets set to non-blocking and everything breaks. I’m unsure exactly where because its a pain to track down.

That said, Jack came to the rescue with this gem:

ansible-galaxy install andrewrothstein.etcd-cluster | cat -

Which unblocks me. It will be interesting to see if other people encounter problems with this change.

,

Simon LyallAudiobooks – October 2022

The Man from the Future: The Visionary Life of John von Neumann by Ananyo Bhattacharya

A good overview of von Neumann’s life and introduction to his most important work. An accessible read that keeps things interesting. 3/5

Sam Walton: Made in America by Sam Walton

Covers the authors life and especially the creation and growth of Walmart. Lots of details about running the business and the industry. 4/5

The Other Side of History: Daily Life in the Ancient World by Robert Garland

48 lectures covering daily life in Egypt, Greece, Rome and Medieval Britain. Plus a few other times & places. Quite interesting. 3/5

We Don’t Need Roads: The Making of the Back to the Future Trilogy by Caseen Gaines

An overview of the making of the movies. Some good stories and I’m sorry it wasn’t longer 3/5

The Hidden Habits of Genius: Beyond Talent, IQ, and Grit—Unlocking the Secrets of Greatness by Craig M. Wright

The “14 key traits of genius, from curiosity to creative maladjustment to obsession”. Some interesting stories but not much really actionable. 2/5

My Audiobook Scoring System

  • 5/5 = Brilliant, top 5 book of the year
  • 4/5 = Above average, strongly recommend
  • 3/5 = Average. in the middle 70% of books I read
  • 2/5 = Disappointing
  • 1/5 = Did not like at all

Share

Dave HallUpgrading to AWS Lambda Powertools for Python v2

Learn how easy it is to upgrade AWS Lambda Powertools to version.

,

Lev LafayetteThe End of Duolingo?

In late 2015 I started using Duolingo, completing sixteen skills trees across ten languages since then. These were mainly, but not exclusively, European languages (including that pan-European auxiliary language, Esperanto). All of which gave some practical use on what were annual trips. In addition to the skill trees I made reasonable progress in Dutch, some progress in Catalan (from Spanish), Czech, and even the trio of Norwegian, Swedish, and Danish (on account of someone saying they were so similar). To say that I am a consistently active user of the application is fair; last year I was rated in the top 0.1% of users in the world.

For several years I have been a paid subscriber to the service at the princely sum of c$100 per year. However, a few days ago, I cancelled my subscription. The reason for this is quite simple; utterly horrendous changes to the user interface. Ignoring the principle of "if it ain't broke, don't fix it", the powers that be at Duolingo have foisted these changes upon a user community that is less than pleased. The Youtube video explaining the changes, at the time of writing, has 116K views; a mere 536 have liked the video and 1.8K had voted it down.

The problems are well-explained by many of the comments on the video; the loss of the self-paced and self-directed learning path in favour of a one-path-only approach, reminiscent of the boardgame, "Candyland", the substantial loss of screen real-estate, and the frustration of trying to reach a lesson of choice, the enforced combination of "lessons" and "stories" (I don't mind this), and the unnecessary animations (I always turned these off in the past). Of the close to 700 comments, almost every single one is negative. It would seem that others too will be ending their subscription, not because of the content of the application, but because of the changes, they made to how people use it.

Certainly, Duolingo has lost people in the past - closing down their forums, ending the language incubators, etc. Those policy changes were annoying, but the comments suggest this is different; this is a visceral hatred, "nerd rage quit" level of disappointment.

This is, of course, not the first time that one has witnessed a mass exodus from an application following radical changes to a user interface. Three years ago Niantic did the same to the game Ingress. When Ubuntu introduced the Unity Desktop, there was a significant switch by users to alternatives. Even when the Luna style was introduced with Windows XP, many users continued to use Windows Classic. One could even give the controversy when Dungeons & Dragons 4th edition was released compared to the 3/3.5 editions - alternative companies, still operating today - made a small fortune by continuing the old system.

The lesson to learn here is that even when usability experts point out numerous benefits to a new interface (e.g., Unity Desktop) or there is an improvement to content, or the marketing people think that making a system similar to other popular products (e.g., D&D 4th edition), a radical change to a user interface system is a hostile attack on the existing user community. The reason for this is deeply part of educational psychology; people engage primarily with content. The user interface is a system to get them to the content. When a user learns a system the process becomes unconscious. Only when the system actively gets in way of the user accessing the content is there a problem, and when that is the case small and incremental changes generate popularity, not rejection.

This is why radical changes in the interface invariably frustrate existing users. What was an unconscious process has to be relearned. If the design actually is more accessible, the learning curve is short, and the access to content is easier, then the period of frustration will be less if users remain with the product. Niantic managed not to bleed to zero Ingress users by allowing users the opportunity to tone down the resource-intensive and overwhelming graphics, for example. The Unity Desktop environment eventually became acceptable to Ubuntu users, as did the Luna style for MS Windows. Duolingo has made a new interface where the design is less accessible, and whilst with a short learning curve (you can only follow one path), access to content is significantly harder.

Duolingo's CEO, Luis von Ahn has made it clear that the "simpler" interface will not be changed and new users must use it now and (almost) all users by the end of October. He is betting that Duolingo's new interface can grow the number of users and operating income of the company. This is probably not going to happen; whilst Duolingo's revenues increased in 2022, the operating income and profits are in the territory of a a $60 million USD loss. Massive investment has been made in the changes, with the hope to reverse a decline in operating income and monthly users.

This is a crash-through or crash approach when a principle of sunk costs should be applied. Perhaps if the backlash from the user community is strong enough and they vote with their feet (and their wallets), the company will revert back to the more popular environment. In the meantime, and as a little hack, there is one way users can keep the old interface; set up a "school" (left-hand column), give it a name, go to "settings", select "older version" and whatever other options you desire (e.g., "multiple languages" taught), and then go back to Duolingo via the left-hand column. Apparently, this will remain in place until the end of the year. After that, there's a golden opportunity for different language applications.

,

Andrew RuthvenLet's Encrypt with Octavia in OpenStack

I like using Catalyst Cloud to host some of my personal sites. In the past I used to use CAcert for my TLS certificates, but more recently I've been using Let's Encrypt for my TLS certificates as they're trusted in all browsers. Currently the LoadBalancer as a Service (LBaaS) in Catalyst Cloud doesn't have built in support for Let's Encrypt. I could use an apache2/nginx proxy and handle the TLS termination there and have that manage the Let's Encrypt lifecycle, but really, I'd rather use LBaaS.

So I thought I'd set about working out how to get Dehydrated (the Let's Encrypt client I've been using) to drive LBaaS (known as Octavia). I figured this would be of interest to other people using Octavia with OpenStack in general, not just Catalyst Cloud.

There's a few things you need to do. These instructions are specific to Debian:

  1. Install and configure Dehydrated to create the certificates for the domain(s) you want.
    • apt install barbican
  2. Create the LoadBalancer (use the API, ClickOps, whatever), just forward port 80 for now (see sample Apache configs below).
  3. Save the sample hook.sh below to /etc/dehydrated/hook.sh, you'll probably need to customise it, mine is a bit more complicated!
  4. Insert the UUID of your LoadBalancer in hook.sh where LB_LISTENER is set.
  5. Create /etc/dehydrated/catalystcloud/password as described in hook.sh
  6. Save OpenRC file from the Catalyst Cloud dashboard as /etc/dehydrated/catalystcloud/openrc.sh
  7. Install jq, openssl and the openstack tools, on Debian this is:
    • apt install jq openssl python3-openstackclient python3-barbicanclient python3-octaviaclient
  8. Add TLS termination to your LoadBalancer
  9. You should be able to rename the latest certs /var/lib/dehydrated/certs/$DOMAIN and then run dehydrated -c to have it reissue and then deploy a cert.

As we're using HTTP-01 Challenge Type here, you need to have the LoadBalancer forwarding port 80 to your website to allow for the challenge response. It is good practice to have a redirect to HTTPS, here's an example virtual host for Apache:

<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias example.com

    RewriteEngine On
    RewriteRule ^/.well-known/ - [L]
    RewriteRule ^/(.*)$ https://www.example.com/$1 [R=301,L]

    <Location />
        Require all granted
    </Location>
</VirtualHost>
You all also need this in /etc/apache2/conf-enabled/letsencrypt.conf:
Alias /.well-known/acme-challenge /var/lib/dehydrated/acme-challenges

<Directory /var/lib/dehydrated/acme-challenges>
        Options None
        AllowOverride None

        # Apache 2.x
        <IfModule !mod_authz_core.c>
                Order allow,deny
                Allow from all
        </IfModule>

        # Apache 2.4
        <IfModule mod_authz_core.c>
                Require all granted
        </IfModule>
</Directory>

And that should be all that you need to do. Now, when Dehydrated updates your certificate, it should update your LoadBalancer as well!

Sample hook.sh:
deploy_cert() {
    local DOMAIN="${1}" KEYFILE="${2}" CERTFILE="${3}" FULLCHAINFILE="${4}" \
          CHAINFILE="${5}" TIMESTAMP="${6}"
    shift 6

    # File contents should be:
    #   export OS_PASSWORD='your password in here'
    . /etc/dehydrated/catalystcloud/password

    # OpenRC file from the Catalyst Cloud dashboard
    . /etc/dehydrated/catalystcloud/openrc.sh --no-token

    # UUID of the LoadBalancer to be managed
    LB_LISTENER='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'

    # Barbican uses P12 files, we need to make one.
    P12=$(readlink -f $KEYFILE \
        | sed -E 's/privkey-([0-9]+)\.pem/barbican-\1.p12/')
    openssl pkcs12 -export -inkey $KEYFILE -in $CERTFILE -certfile \
        $FULLCHAINFILE -passout pass: -out $P12

    # Keep track of existing certs for this domain (hopefully no more than 100)
    EXISTING_URIS=$(openstack secret list --limit 100 \
        -c Name -c 'Secret href' -f json \
        | jq -r ".[]|select(.Name | startswith(\"$DOMAIN\"))|.\"Secret href\"")

    # Upload the new cert
    NOW=$(date +"%s")
    openstack secret store --name $DOMAIN-$TIMESTAMP-$NOW -e base64 \
        -t "application/octet-stream" --payload="$(base64 < $P12)"

    NEW_URI=$(openstack secret list --name $DOMAIN-$TIMESTAMP-$NOW \
        -c 'Secret href' -f value) \
        || unset NEW_URI

    # Change LoadBalancer to use new cert - if the old one was the default,
    # change the default. If the old one was in the SNI list, update the
    # SNI list.
    if [ -n "$EXISTING_URIS" ]; then
        DEFAULT_CONTAINER=$(openstack loadbalancer listener show $LB_LISTENER \
            -c default_tls_container_ref -f value)

        for URI in $EXISTING_URIS; do
            if [ "x$URI" = "x$DEFAULT_CONTAINER" ]; then
                openstack loadbalancer listener set $LB_LISTENER \
                    --default-tls-container-ref $NEW_URI
            fi
        done

        SNI_CONTAINERS=$(openstack loadbalancer listener show $LB_LISTENER \
            -c sni_container_refs -f value | sed "s/'//g" | sed 's/^\[//' \
            | sed 's/\]$//' | sed "s/,//g")

        for URI in $EXISTING_URIS; do
            if echo $SNI_CONTAINERS | grep -q $URI; then
                SNI_CONTAINERS=$(echo $SNI_CONTAINERS | sed "s,$URI,$NEW_URI,")
                openstack loadbalancer listener set $LB_LISTENER \
                    --sni-container-refs $SNI_CONTAINERS
            fi
        done

        # Remove old certs
        for URI in $EXISTING_URIS; do
            openstack secret delete $URI
        done
    fi
}

HANDLER="$1"; shift
#if [[ "${HANDLER}" =~ ^(deploy_challenge|clean_challenge|sync_cert|deploy_cert|deploy_ocsp|unchanged_cert|invalid_challenge|request_failure|generate_csr|startup_hook|exit_hook)$ ]]; then
if [[ "${HANDLER}" =~ ^(deploy_cert)$ ]]; then
    "$HANDLER" "$@"
fi

,

Dave HallTracking Infrastructure with SSM and Terraform

Use AWS SSM Parameter Store to share resource references with other teams.

,

Tim RileyOpen source status update, September 2022

Hello there, friends! This is going to be a short update from me because I’m deep in the throes of Hanami 2.0 release preparation right now. Even still, I didn’t want to let September pass without an update, so let’s take a look.

A story about Hanami::Action memory usage

September started and ended with me looking at the r10k memory usage charts for hanami-controller versus Rails. The results were surprising!

Initial memory usage for Hanami::Action vs Rails

We’d been running some of these checks as part of our 2.0 release prep, the idea being that it’d help us shake out any obvious performance improvements we’d need to make. And it certainly did in this case! Hanami (just like its dry-rb underpinnings) is meant to be the smaller and lighter framework; why were we being outperformced by Rails?

To address this I wrote a simple memory profile script for Hanami::Action inheritance (now checked in here) and started digging.

Here were there initial results:

Total allocated: 184912288 bytes (1360036 objects)
Total retained:  104910880 bytes (780031 objects)

allocated memory by gem
-----------------------------------
  56242240  concurrent-ruby-1.1.10
  53282480  dry-configurable-0.15.0
  34120000  utils-8585be837309
  30547488  other
  10720080  controller/lib

That’s 185MB allocated for 10k subclasses, with concurrent-ruby, dry-configurable and hanami-utils being the top three gems allocating memory.

This led me straight to dry-configurable, and after a couple of weeks of work, I arrived at this PR, separating our storage of setting definitions from their configured values, among other things. This change allows us to copy less data at the moment of class inheritance, and in the case of a dry-configurable-focused memory profile, cut the allocated memory by more than half.

From there, I moved back into hanami-controller and updated it to use dry-configurable for all of its inheritable attributes (some were handled separately), also taking advantage the support for custom config classes that Piotr added so we could preserve Hanami::Action’s existing configuration API.

This considerably improved our benchmark! Behold:

Total allocated: 32766232 bytes (90004 objects)
Total retained:  32766232 bytes (90004 objects)

allocated memory by gem
-----------------------------------
  21486072  other
  10880120  dry-configurable-0.16.1
    400040  3.1.2/lib

Yes, we brought 185MB allocated memory down to 33MB! This also brought us on par with Rails in the extreme end of the r10k memory usage benchmark:

Updated memory usage for Hanami::Action vs Rails

Here’s a thing though: the way r10k generates actions for its Rails benchmark is to create a single controller class with a method per action. So for the point on the far right of that chart, that’s a single class with 10k methods. Hardly realistic.

So I made a quick tweak to see how things would look if the r10k Rails benchmark generated a class per endpoint like we do with Hanami::Action:

Hanami::Action vs Rails with a separate controller class per action

That’s more like it. This is another extreme, however: more realistically, we’d see Rails apps with somewhere between 5-10 actions per controller class, which would lower its dot a little in that graph. In my opinion this would be a useful thing to upstream into r10k. It’s already a contrived benchmark, yes, but it’d be more useful if it at least mimicked realistic application structures.

Either way, we finished the month much more confident that we’ll be delivering on our promise of Hanami as the lighter, faster framework alternative. A good outcome!

Along the way, however, things did feel bleak at times. I wasn’t confident that I’d be able to make things right, and it didn’t feel great to think we might’ve spent years putting somethign together that wasn’t going to be able to deliver on some of those core promises. Luckily, I found all the wins we needed, and learnt a few things along the way.

Hanami 2.0, here we come

What else happened in September? Possibly the biggst thing is that we organised ourselves for the runway towards the final Hanami 2.0.0 release.

We want to do everything possible to make sure the release happens this year, so I spent some time organising the remaining tasks on our Trello board into date-based lists, aiming for a release towards the end of November. It looked achievable! The three of us in the core team re-committed ourselves to doing everything we could to complete these tasks in our estimated timeframes.

So far, things have gone very well!

Hanami 2.0.0 release progress on Trello

We’ve all been working tremendously hard, and so far, this has let us keep everything to the schedule. I’ll have a lot to share about our work across October, but that’s all for next month’s update. So in the meantime, I have to put my head back down and get back to shipping a framework. See you all again soon!

Lev LafayetteBorderline Personality Disorder: A Summary

This is a summary of what I have learned over the past twothree years, after my first direct encounter with what is called Borderline Personality Disorder (BPD). Whilst I do not have BPD (although everyone is a little bit on every mental health continuum), I do endeavour to be a loyal and committed ally of people with BPD (pwBPD). In a very real sense, I wish I knew then what I do now; but at least I have made the effort to learn. I hope that these notes are useful to others. For anyone who wishes to be a sincere ally (a catch-all term that should include partners, family, and friends) of a person with BPD it is absolutely necessary to make the effort to listen to the pwBPD and to educate yourself using scholarly sources. Not making the effort means that you're not an ally, regardless of how close you think you are to the person, and not using scholarly sources will cause more harm and prejudice than good.

This document was initially written at the end of BPD Awareness Week 2022 in Australia and for World Mental Health Day, and will be updated as new information comes to hand. Throughout all the content here it is emphasised that (a) quantifiers are always required (many, most, some, etc) and every BPD person is unique and will not show all characteristics and (b) always see the person. Please note that I am not a psychologist, although I am a student of the subject and have completed most of a Graduate Diploma of Applied Psychology at the University of Auckland. I encourage people to donate to the Australian BPD Foundation.

Warning: This article mentions suicide, self-harm, and abuse.

Last update: April 18, 2023

Definition and Prevalence

"Borderline Personality Disorder" is a mental health condition marked by a long-term pattern of intense emotional reactions, divergent moods, unstable interpersonal relations, impulsivity, and issues in self-identity and self-direction. The term itself was coined when the condition of behaviours was deemed to be on the borderline of psychosis (difficulty in determining what is real) and neuroticism (disorders that cause constant distress), where a neurotic person in a time of stress would show signs of psychosis. Whilst neither 'psychosis' nor 'neuroticism' are used as formal mental health descriptors, the term "borderline" has stuck. The term was included in DSM-III (1980) where it remains to the current edition. An alternative, and more intuitive term, is "Emotionally unstable personality disorder" (EUPD).

The median prevalence of BPD is c1% (Ellison et al, 2018). In clinical settings, BPD prevalence is around 10-12% in outpatient psychiatric clinics and 20-22% among inpatient clinics. Prevalence is notably higher among incarcerated individuals and notably lower among the elderly. There is a pronounced gender distinction with women diagnosed over men at a 3:1 ratio (Skodol, Bender, 2003). Underdiagnosis and misdiagnosis are unfortunately common, with over 40% of pwBPD had been previously misdiagnosed with other disorders like bipolar disorder or major depressive disorder (Ruggero et al, 2010).

Diagnosis and Symptoms

The DSM-5 (p663, 2013) gives the following as diagnostic criteria. Formal diagnosis requires satisfying of five or more of the criteria.

1. Frantic efforts to avoid real or imagined abandonment (Note: Do not include suicidal or self-mutilating behaviour covered in Criterion 5)
2. A pattern of unstable and intense interpersonal relationships characterised by alternating
between extremes of idealisation and devaluation
3. Identity disturbance: markedly and persistently unstable self-image or sense of self
4. Impulsivity in at least two areas that are potentially self-damaging (e.g. spending, sex, substance abuse, reckless driving, binge eating) (Note: Do not include suicidal or self-mutilating behaviour covered in Criterion 5)
5. Recurrent suicidal behaviour, gestures, or threats, or self-mutilating behaviour
6. Affective instability due to a marked reactivity of mood (e.g. intense episodic dysphoria, irritability or anxiety usually lasting a few hours and only rarely more than a few days)
7. Chronic feelings of emptiness
8. Inappropriate, intense anger or difficulty controlling anger (e.g. frequent displays of temper,
constant anger, recurrent physical fights)
9. Transient, stress-related paranoid ideation or severe dissociative symptoms

There are similar criteria for the International Classification of Diseases (11th Revision) which describes "the borderline pattern descriptor" as follows:

A pervasive pattern of instability of interpersonal relationships, self-image, and affects, and marked impulsivity, as indicated by many of the following:

1. Frantic efforts to avoid real or imagined abandonment
2. A pattern of unstable and intense interpersonal relationships
3. Identity disturbance, manifested in markedly and persistently unstable self-image or sense of self
4. A tendency to act rashly in states of high negative affect, leading to potentially self-damaging behaviours
5. Recurrent episodes of self-harm
6. Emotional instability due to marked reactivity of mood
7. Chronic feelings of emptiness
8. Inappropriate intense anger or difficulty controlling anger
9. Transient dissociative symptoms or psychotic-like features in situations of high affective arousal

If one thinks that they fit the criteria for BPD it is essential to seek a professional diagnosis. Without professional treatment, one is taking an enormous risk of harm to themselves and others. Likewise, if one thinks that another person fits the criteria, raise the matter very gently and delicately with a motivation of care and with recognition and self-awareness that you are not a professional.

Causes and Neurology

Borderline personality disorder often begins in adolescence or early adulthood. It is characterized by problems with interpersonal relationships (they are intense, alternating between idealization and devaluation), mood (depression and especially inappropriate, intense anger), and unstable self-image. Current estimates of the general population prevalence of borderline personality disorder range up to 5.9 percent, and recent studies of college students suggest that up to 17 percent struggle with significant borderline traits. Borderline personality disorder is associated with psychiatric disability, substance abuse, eating disorders, and medical problems. BPD patients showed significantly higher scores on both primary and secondary global rates of psychopathic behaviour associated with patterns of executive dysfunction (López-Villatoro et al, 2020)

The heritability of BPD is between 37% to 69%, a rather wide range (Gunderson et al, 2011), with indications that is one of the most heritable disorders (Torgersen et al, 2000). However, even when researchers do note specific linkages to genetics variation between genetic and environmental factors are balanced at 42%/58% (Distel et al, 2008). These environmental factors are commonly associated with the result of childhood trauma such as neglect and abuse; there is little doubt that a person who has experienced childhood trauma is at an increased risk for developing BPD and PTSD (Cattane et al 2017).

Real-time brain imaging scans have established that pwBPD are physically unable to regulate emotions (Nauret, 2017). Neuroimaging shows that pwBPD typically has a reduction in the brain's regions that regulate stress responses, emotions, and decision-making including the amygdala, the hippocampus, and the orbitofrontal cortex (O'Neill, Frodl, 2012). There is dysregulation of the hypothalamic-pituitary-adrenal axis, responsible for the production of cortisol, released during times of stress; pwBPD have abnormal levels of cortisol production (Cattane, et al 2017), reflected in damage erosion of the very areas of the brain responsible for stress regulation and decision-making. Amygdala damage is associated with impulsive behaviour, a lessens aversion to risk and loss (Gupta et al 2011), and also with hypervigilance (Terburg et al, 2012). Damage to the amygdala (emotional processing) and the hippocampus (declarative of episodic recollection) also reduces the capacity for memory (Yang, Wang, 2017). These all contribute to BPD being described as the mental illness with the highest level of psychological pain.

Comorbidities

There are a number of comorbidities with BPD. The following are a few words on the most common, including Eating Disorders, Attention Deficit Hyperactivity Disorder, (complex and chronic) Post-Traumatic Stress Disorder, Narcissistic Personality Disorder, and Bipolar Disorders

Eating disorders and BPD are co-morbid, with some 53.8% co-occurrence from one extensive study (Zanarini, et al 2010), compared to 24.6% of patients with other personality disorders and more specifically, 21.7% of patients with BPD met criteria for anorexia nervosa and 24.1% for bulimia nervosa. Like other co-morbidities, an association has been drawn between eating disorders, BPD, and the environmental factor of childhood trauma, whether in the form of neglect or abuse (Sansone, Sansone, 2017).

Attention deficit hyperactivity disorder (ADHD) and BPD, is another frequent comorbidity, in a clinical setting, ranging from 16.1 to 38% of BPD patients (Weiner et al, 2019). This comorbidity questions whether it is appropriate to view either as an entirely early-onset neurological disorder (ADHD) or a later-onset environmental disorder (BPD). As with many other comorbidities, the expression of characteristics is more severe, hence people with ADHD and BPD are even more impulsive than those with BPD alone, and with a higher level of emotional dysregulation than those with ADHD alone.

Post-traumatic stress disorder (PTSD) including complex PTSD, and borderline personality disorder commonly co-occur, approximately 25-30% (Pagura et al 2010, Frías and Palma, 2015) of the time. Whilst PTSD is characterised by (a) a sense of threat, (b) avoidance, and (c) re-experiencing. complex PTSD has, in addition, (d) interpersonal avoidance and difficult interpersonal relationships (e) negative self-concept, and (f) affective instability. BPD does not have (a), (b), and (c), but does have (d), (e), (f) and, in addition, (g) anger, (h) chronic emptiness, (i) self-injury behaviours (j) transient psychotic and dissociation and (k) fear of abandonment. Individuals with comorbid PTSD-BPD have a poorer quality of life on average, with higher levels of self-harm.

Narcissistic Personality Disorder (NPD) and Borderline Personality Disorder (BPD) are both "cluster B" disorders, characterised by dramatic and intense behaviour (at least to observers), and impulsive behaviour. This cluster includes NPD, BPD, anti-social personality disorder, histrionic personality disorder, etc In addition to this general overlap the co-occurrence of BPD and NPD has been assessed from a range of 13% (Hörz-Sagstetter et al 2018) to 39% (Grant et al, 2008). There is a possibility that it is particularly associated with "vulnerable narcissism", whose traits include hypersensitivity, defensiveness, and low self-esteem. People with NPD and BPD are less likely to see a remission of BPD as NPD people have a lower motivation to seek therapy, and NPD is very difficult to treat (Caligor et al, 2015).

Bipolar Disorders and BPD also occur, in approximately 20% of cases (Zimmerman, 2019) and there is an ongoing discussion on whether BPD should be part of the bipolar spectrum, although most recent literature suggests that they are distinct, and the debate has actually sidetracked from the substantive issue. Like other comorbid states, people with "borderpolar" have higher levels of impaired functioning, substance abuse disorders, and self-harm (Patel et al, 2019). Further, people with BPD and a Bipolar Disorder are more likely to have PTSD as well, generating an especially challenging combination that has been insufficiently researched and is likely underdiagnosed.

Prognosis

BPD conditions remain throughout the lifespan, although with variations in symptoms (Biskin, 2015). In some cases BPD symptoms can be observed in childhood, however, there is an absence of evidence regarding the course of development of those who do not meet the full criteria. Adolescence is usually when BPD is recognised, although there is evidence of remission in follow-up studies ranging from 40% to 65%, although residual symptoms are not always predictable. Adult BPD longitudinal studies also suggest a gradual decline in symptoms, with periods of remission of recurrence. The decline of symptoms was mainly in the behavioural aspects of impulsivity; self-harm and suicide remained a factor with one large study indicating a 10% suicide rate after 27 years of follow-up, mainly patients in their 30s with multiple failed treatments. Even with a decline in symptoms over time functional recovery - defined as remission along with full-time vocational or educational activity and at least one stable and supportive relationship with a close friend or partner - occurred in only just more than 50% of patients (Zanarini et al, 2012).

People with BPD have a reduced life expectancy ranging from 14 to 27.5 years, with a median value of 20 (Castle, 2019). Most of the early mortality is largely due to cardiovascular deaths with major risk factors (e.g., obesity, smoking, poor diet, and lack of exercise) significantly greater among people with BPD. Other notable risk factors include arteriosclerosis, hypertension, hepatic disease, arthritis, gastrointestinal disease, cardiovascular disease, and sexually transmitted diseases. These can be attributed to maladaptive lifestyle choices (smoking, drugs, alcohol, diet) as well as iatrogenic (prescription medicines). This is hardly helped by chronic sleep issues (Selby, 2013). The problems are often compounded with a person with BPD having comorbidity, and also by the stigma attached by BPD even in the responses of health professionals. Suicide rates vary from up to 10% of cases from follow-back research, or from 3-6% in prospectively followed cohorts, and most occur later in life (mean age of 37, standard deviation of 10) (Paris, 2019)

Treatment

There is no cure for BPD, but recovery and management are possible. There have only been very modest evidence of neurogenesis of the amygdala (Jhaveri, 2018) and mood disorders are known to weaken the prospect of neurogenesis of the hippocampus. In other words, the very experience of having BPD reduces the possibility of recovery from BPD (Toda et al, 2019). There is evidence that deep brain stimulation can help relieve some psychological and behavioral side effects, such as hypervigilance (Langevin, 2012).

There are some regularly prescribed medications for pwBPD, typically antipsychotics (Grootens, Verkes, 2005) and mood stabilisers (Lieb et al, 2010). Usually, psychotherapy has been shown to be particularly beneficial, with Dialectical Behaviour Therapy (DBT) offering the greatest rates of success (Choi-Kain, 2017). It is, of course, not something that necessarily works for everybody with BPD, and other therapies may be more appropriate depending on the individual (e.g., schema therapy, mentalisation-based treatment, transference-focussed psychotherapy). A particular warning is raised for matters of misdiagnosis, especially with common co-morbidities such as PTSD. In many cases, a treatment that is very effective for PTSD can aggravate BPD and vice-versa e.g., trauma history, mood swings, and alienation from others (Hammond, 2020)

Community

The BPD Community consists of people with lived experience of Borderline Personality Disorder. This can include people with the illness (diagnosed, undiagnosed, treated, and in remission), their close friends, family, partners, and allies. The following are a few short comments on the lived experiences of both pwBPD and those in their life. This section of the summary is somewhat more informal than what has preceded.

Availability, Understanding, Solutions

A common error by allies have when a pwBPD is having an episode of extreme emotions (anger, sadness, anxiety, etc) is that they seek to provide rational solutions to what is a perceived problem. This may be a genuine response motivated by care and love, but it is not the appropriate approach. A person in such a situation is experiencing an emotional disturbance and the experience must also be dealt with emotionally. People with BPD have at least equal and often heightened levels of emotional empathy, but their emotional cognition and performance are quite poor (Niedtfeld, 2017). This is often referred to as "the Borderline Empathy Paradox", where it is common of pwBPD to detect even subtle emotional states of others the also typically have serious deficits of cognitive and behavioural empathy (Salgado et al 2020).

The following steps - availability, understanding, and solutions - must be carried out in order with each step depending on the preceding. An alternative name for these is "SET theory" (Kreisman, 2018), standing for "Support, Empathy, Truth", although that will get confusing for people with an interest in discrete mathematics (such as the author).

Availability: One should recall that a pwBPD suffers a chronic fear of abandonment; thus availability must be of the first priority. Simply being present can help alleviate the fear. Statements of support and engagement are also valuable: "I am here for you", "I care about you", "I want to help", etc.

Understanding: Once availability is established, the pwBPD is likely to express their feelings. Their ally must display empathy and understanding at this point. The pwBPD may seek to ground their feelings in events or interpretations that might be completely erroneous, conflated, etc. The ally should not seek to correct them or downplay the real or imagined causes, but rather validate the emotions. This requires some attention on the behalf of the ally to listen to the feelings as well as the words being used. Feelings are ALWAYS valid, even if the reasons are not and the pwBPD feels their feelings more viscerely than anyone else. The ally should give statements that validate the feelings: "This must be very frustrating for you".

Solutions: Only once the pwBPD has an assurance of an ally's availability, and the empathic rapport of understanding and validating their emotional state is established, should potential solutions be offered. These need to be factual or based on the ally's commitments (and the ally had better follow through): "This is what I can do to help", "If you do x, then y will happen. Perhaps consider z", etc.

Shame, Guilt, and Remorse

It is virtually a given that pwBPD will engage in words or actions that are very hurtful and damaging to those close to them. Unlike people who have limited emotional range or capacity, a pwBPD feels emotions, including shame and guilt, intensely. The coping mechanisms and responses of pwBPD however are typically very poor and they will often hold on to shame and guilt in a manner that is damaging to their self-esteem (through self-loathing), despair (avoiding establishing commitment through fear of hurting people in the future), or even various of self-harm. For pwBPD it is essential that they learn to turn shame into guilt and guilt into remorse, otherwise the pain will be ongoing.

Shame is more prevalent among pwBPD than guilt (Peters, Geiger 2016). Shame reflects the individual's negative self-concept and self-loathing, and represents the accumulated negative beliefs that the individual has toward themselves. With pwBPD it is an important contributor to anger, unstable mood, instability of interpersonal relationships, externalisation of blame, and self-harm. When a pwBPD is triggered by events that generate shame, including the results of their own impulsivity and other behaviours. However, it is necessary for a pwBPD to develop guilt about actions rather than accumulating further shame about them. Guilt at least focuses on the event and identifies the need to change behaviour, rather than adding to the negative self-image of shame which is an interrnalised and private pain.

The difference between guilt and remorse involves taking ownership of what a person has done that has hurt another. Contact, even indirect contact if necessary, of those that have been impacted is suggested. Informing the wronged party that one feels that they have wronged them and that one is changing themself so it doesn't happen again, is required. Further, asking if there's anything that can be done to make amends is a full acceptance of responsibility. There is no onus on the wronged party to give forgiveness or to accept any offer of amends. However, in most cases, people are forgiving when they see a genuine attempt in a person to change.

Mirroring, Splitting, Discarding, Reconnection

Borderline
Feels like I'm going to lose my mind
You just keep on pushing my love
Over the borderline
-- Madonna, Borderline (1983)

The famous Madonna song, for what it's worth, actually is not about BPD, but for those in a close relationship with a pwBPD, they experience of having one's love "pushed" and that the close ally may lose their own mind is a very common experience. A common descriptor used by both pwBPD and their loved ones is that the experience is like being on an emotional roller-coaster. Many of those who have experienced a relationship with a pwBPD describe a cycle of behaviour that, constitutes manipulative abuse (Brüne, 2016). The actions carried out by a pwBPD are unconsciously driven as they desperately fear abandonment whilst at the same time having high levels of chronic mistrust and a belief that they are unlovable, and a lack of object constancy with their loved ones (Matejko, 2022).

A typical cycle will consist of an initial and often incredible connection between the pwBPD and their loved one, their "favourite person" as described in the culture. The pwBPD will engage in "mirroring", elevating their loved one, affirming their beliefs, dreams, and activities, and will present themselves as exciting and adventurous in the process. For the loved one, they will often describe the experience in highly romantic terms, such as finally meeting their soulmate. This experience, however, does not last; the inevitable flaws of the loved one and the affective instability of the pwBPD will usually mean that the pwBPD will engage in "splitting" against their loved one. Where once they were exalted, they are now treated with equivalent disdain (often with rage and vitriol), and will soon be discarded. During the negative side of the split, the pwBPD, with far greater frequency than others, will establish a new love interest (Michael, 2021). "Splitting" itself is a malformed defense mechanism on the part of the person with BPD (Fetruck et al, 2018) where they convince themselves of the validity of the impending discard.

With the new love interest, the same process is very likely to repeat itself. Often enough, the pwBPD will then engage in the same process and re-establish connection with their original partner with a similar level of original elevation, and the cycle will repeat, or they will find an entirely new relationship; perhaps unsurprisingly, pwBPD tend to have a larger number of romantic relationships over their lifetime (Navarro-Gómez et al, 2017). Assuming a return to the original interest, it is not uncommon for loved ones of a pwBPD to describe how, over a number of years, their partners have discarded them several times and more. Whilst patience and commitment are admirable in any relationship, they will be insufficient in this situation. Therapy for pwBPD and couples therapy for the pwBPD and their partner is also required for success. Establishing clear boundaries and agreed consequences for particular actions should also assist.

Lying, Gaslighting, Lovebombing

The perception of reality for a pwBPD is driven by the current emotional state, which is subject to heightened levels of intensity and instability. As a direct consequence of this, a pwBPD engages in activities that, to an outsider, are like lying or gaslighting but are driven by fearful states rather than an act of malicious deception - it's more an act of desperation, rather than malice. For example, pwBPD often have a weakened level of promissory commitment to the expressions that they provide. Their reality is very much in the "here and now", rather than in the longer term, even when expressed in those terms. At the moment a pwBPD will quite sincerely and wholeheartedly believe what they are saying but will either forget the content entirely or have a radical change in affective orientation. Reminding the pwBPD of their prior commitments is important, but even more can be gained with a reminder of the emotional content of the commitment.

Another result of this emotional, rather than factual, perception is that pwBPD present statements that seem like gaslighting. Emotionally healthy people will develop feelings based on facts. However, pwBPD may unconsciously revise the facts to fit their current feelings or invent facts to fill in memory gaps. Tragically, this behaviour also weakens the ability of the pwBPD to develop a coherent autobiographical sense of self or firm memories. This can also be very confronting to an ally, whose immediate reaction will be to correct the factual error; this is a mistake and instead, the same SET principles described previously should be applied; the facts are secondary; empathy and understanding of the feeling must have priority.

Another experience that loved ones of a pwBPD experience is "lovebombing". These are overwhelming displays of affection and attachment. As the Oxford English Dictionary states: "the action or practice of lavishing someone with attention or affection, especially in order to influence or manipulate them". For a pwBPD the lavishing is real, at the moment. They are not consciously trying to manipulate their loved one. They are, in fact, both terrified of losing their loved one (thus the ovewhelming display of affection) and, at the same time, ready to engage in a "protective discard" on the assumption that their loved one will leave them, and equally fearful of engulfment. Love-bombing can be seen as a symptom of an insecure attachment style, that matches with 90%+ of pwBPD, to the point that is considered almost tautological (Kaurin et al, 2020), and the disorganized insecure attachment style in particular (Agrawal et al, 2004).

Concluding Remarks

This summary is a compilation of my own notes and research over the past twothree years or so. It really is a personal essay, albeit written with my own tendency to an academic style, to make sense of what is a common and often debilitating mental illness. Despite the various difficulties, emphasis is again placed on the importance of individual variation with pwBPD, the legitimacy of their voice in explaining the lived experience of the condition, and the fact that the person who has BPD is also so much more than the illness that they carry. There are a terrible stigma (Aviram, et al, 2006) attached to pwBPD in popular culture and the media, and there are prejudices that abound and most surprisingly in the professions that should be the most helpful. Of course, pwBPD are just as prone to engaging in consciously hurtful acts toward others as anyone else, but in the main, they are incredibly empathic and caring although often unable to fully control their impulses. Genuine sympathy, understanding, and treatment all will help make life much better for them and us.

References

Agrawal, H. R., Gunderson, J., Holmes, B. M., & Lyons-Ruth, K. (2004). Attachment studies with borderline patients: a review. Harvard review of psychiatry, 12(2), 94–104. https://doi.org/10.1080/10673220490447218

Aviram RB, Brodsky BS, Stanley B (2006). "Borderline personality disorder, stigma, and treatment implications". Harvard Review of Psychiatry. 14 (5): 249–256. doi:10.1080/10673220600975121

Biskin RS. (2015). The Lifetime Course of Borderline Personality Disorder. Can J Psychiatry. 2015 Jul;60(7):303-8. doi: 10.1177/070674371506000702. PMID: 26175388; PMCID: PMC4500179.

Brüne M. Borderline Personality Disorder: Why 'fast and furious'?. (2016) Evol Med Public Health;2016(1):52–66. doi:10.1093/emph/eow002

Caligor E, Levy KN, Yeomans FE. (2015). Narcissistic personality disorder: Diagnostic and clinical challenges. AJP. 2015;172(5):415-422. doi:10.1176/appi.ajp.2014.14060723

Castle, D. J. (2019). The complexities of the borderline patient: how much more complex when considering physical health?. Australasian Psychiatry, 27(6), 552-555.

Cattane N, Rossi R, Lanfredi M, Cattaneo A. (2017). Borderline personality disorder and childhood trauma: exploring the affected biological systems and mechanisms. BMC Psychiatry. 2017;17(1):221. doi:10.1186/s12888-017-1383-2

Choi-Kain LW, Finch EF, Masland SR, Jenkins JA, Unruh BT. (2017). What works in the treatment of borderline personality disorder. Curr Behav Neurosci Rep. 2017;4(1):21-30. doi:10.1007/s40473-017-0103-z

Distel et al. Heritability of borderline personality disorder features is similar across three countries. Psychological Medicine, 2008; 38 (9): DOI: 10.1017/S0033291707002024

Ellison WD, Rosenstein LK, Morgan TA, Zimmerman M. (2018). Community and Clinical Epidemiology of Borderline Personality Disorder. Psychiatr Clin North Am. 2018 Dec;41(4):561-573. doi: 10.1016/j.psc.2018.07.008. Epub 2018 Oct 16. PMID: 30447724.

Fertuck EA, Fischer S, Beeney J. (2018). Social cognition and borderline personality disorder: Splitting and trust impairment findings. Psychiatr Clin North Am. 2018;41(4):613-632 doi:10.1016/j.psc.2018.07.003

Frías Á, Palma C. Comorbidity between post-traumatic stress disorder and borderline personality disorder: A review. Psychopathology. 2015;48(1):1-10. doi:10.1159/000363145
https://pubmed.ncbi.nlm.nih.gov/25227722/

Grant BF, Chou SP, Goldstein RB, et al. Prevalence, correlates, disability, and comorbidity of DSM-IV borderline personality disorder: Results from the Wave 2 National Epidemiologic Survey on Alcohol and Related Conditions. J Clin Psychiatry. 2008;69(4):533-545. doi:10.4088/jcp.v69n0404

Grootens KP, Verkes RJ. (2005). "Emerging evidence for the use of atypical antipsychotics in borderline personality disorder". Pharmacopsychiatry. 38 (1): 20–3. doi:10.1055/s-2005-837767.

Gunderson JG, Zanarini MC, Choi-Kain LW, Mitchell KS, Jang KL, Hudson JI (August 2011). "Family Study of Borderline Personality Disorder and Its Sectors of Psychopathology". JAMA: The Journal of the American Medical Association. 68 (7): 753–762. doi:10.1001/archgenpsychiatry.2011.65.

Gupta R, Koscik TR, Bechara A, Tranel D. The amygdala and decision-making. (2011). Neuropsychologia. 2011 Mar;49(4):760-6. doi: 10.1016/j.neuropsychologia.2010.09.029. Epub 2010 Oct 8. PMID: 20920513; PMCID: PMC3032808.

Hammond, C. (rev) (2020). How PTSD can look like Borderline Personality Disorder. Psych Central.

Hörz-Sagstetter S, Diamond D, Clarkin JF, et al. (2018) Clinical characteristics of comorbid narcissistic personality disorder in patients with borderline personality disorder. J Pers Disord. 2018;32(4):562-575. doi:10.1521/pedi_2017_31_306

Jhaveri, D. (2018). Neurogenesis in the emotion-processing centre of the brain. Australasian Science, 39(1), 24-26.

Kaurin, A., Beeney, J. E., Stepp, S. D., Scott, L. N., Woods, W. C., Pilkonis, P. A., & Wright, A. G. C. (2020). Attachment and Borderline Personality Disorder: Differential Effects on Situational Socio-Affective Processes. Affective science, 1(3), 117–127. https://doi.org/10.1007/s42761-020-00017-7

Kreisman JJ. (2018). Talking to a Loved One with Borderline Personality Disorder, Communication Skills to Manage Intense Emotions, Set Boundaries, and Reduce Conflict. New Harbinger Publications.

Langevin JP. (2012). The amygdala as a target for behavior surgery. Surg Neurol Int. 2012;3(Suppl 1):S40-6. doi: 10.4103/2152-7806.91609. Epub 2012 Jan 14. PMID: 22826810; PMCID: PMC3400485.

Lieb, Klaus; Völlm, Birgit; Rücker, Gerta; Timmer, Antje; Stoffers, Jutta M. (2010). "Pharmacotherapy for borderline personality disorder: Cochrane systematic review of randomised trials". The British Journal of Psychiatry. 196 (1): 4–12. doi:10.1192/bjp.bp.108.062984

López-Villatoro, J. M., Diaz-Marsá, M., Mellor-Marsá, B., De la Vega, I., & Carrasco, J. L. (2020). Executive dysfunction associated with the primary psychopathic features of borderline personality disorder. Frontiers in Psychiatry, 11, 514905.

Matejko, S. (2022). Understanding Object Constancy in Borderline Personality Disorder and Narcissism, PsychCentral, 2022
https://psychcentral.com/disorders/borderline-personality-disorder/objec...

Michael J, Chennells M, Nolte T, et al (2021). Probing commitment in individuals with borderline personality disorder. J Psychiatric Res. 2021;137:335-341. doi:10.1016/j.jpsychires.2021.02.062

Nauert, R., (2017). Brain Scans Clarify Borderline Personality Disorder. PsychCentral
https://psychcentral.com/news/2017/09/04/brain-scans-clarify-borderline-...

Navarro-Gómez S, Frías Á, Palma C. Romantic relationships of people with borderline personality: A narrative review. (2017) PSP. 2017;50(3):175-187. doi:10.1159/000474950

Niedtfeld I. (2017) Experimental investigation of cognitive and affective empathy in borderline personality disorder: effects of ambiguity in multimodal social information processing. Psychiatry Res 253:58–63. doi: 10.1016/j.psychres.2017.03.037

O'Neill A, Frodl T (October 2012). "Brain structure and function in borderline personality disorder". Brain Structure & Function. 217 (4): 767–782. doi:10.1007/s00429-012-0379-4

Pagura, J., Stein, M. B., Bolton, J. M., Cox, B. J., Grant, B., & Sareen, J. (2010). Comorbidity of borderline personality disorder and posttraumatic stress disorder in the US population. Journal of psychiatric research, 44(16), 1190-1198.

Paris J. (2019). Suicidality in Borderline Personality Disorder. Medicina (Kaunas). 2019 May 28;55(6):223. doi: 10.3390/medicina55060223. PMID: 31142033; PMCID: PMC6632023.

Patel RS, Manikkara G, Chopra A. Bipolar Disorder and Comorbid Borderline Personality Disorder: Patient Characteristics and Outcomes in US Hospitals. (2019) Medicina (Kaunas). 2019 Jan 14;55(1):13. doi: 10.3390/medicina55010013. PMID: 30646620; PMCID: PMC6358827.

Peters JR, Geiger PJ. (2016). Borderline personality disorder and self-conscious affect: Too much shame but not enough guilt? Personal Disord. 2016 Jul;7(3):303-8. doi: 10.1037/per0000176. Epub 2016 Feb 11. PMID: 26866901; PMCID: PMC4929016.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4929016/

Ruggero CJ, Zimmerman M, Chelminski I, Young D. Borderline personality disorder and the misdiagnosis of bipolar disorder. (2010). J Psychiatr Res. 2010;44(6):405–408. doi:10.1016/j.jpsychires.2009.09.011

Salgado, R. M., Pedrosa, R., & Bastos-Leite, A. J. (2020). Dysfunction of Empathy and Related Processes in Borderline Personality Disorder: A Systematic Review. Harvard review of psychiatry, 28(4), 238–254. https://doi.org/10.1097/HRP.0000000000000260

Sansone RA, Sansone LA. Childhood trauma, borderline personality, and eating disorders: a developmental cascade. Eat Disord. 2007;15(4):333-46. doi:10.1080/10640260701454345

Selby E. A. (2013). Chronic sleep disturbances and borderline personality disorder symptoms. Journal of consulting and clinical psychology, 81(5), 941–947. https://doi.org/10.1037/a0033201

Skodol AE, Bender DS. (2003) Why are women diagnosed borderline more than men? Psychiatr Q. 2003 Winter;74(4):349-60. doi: 10.1023/a:1026087410516. PMID: 14686459.

Terburg D, Morgan BE, Montoya ER, Hooge IT, Thornton HB, Hariri AR, Panksepp J, Stein DJ, van Honk J. Hypervigilance for fear after basolateral amygdala damage in humans. (2012). Transl Psychiatry. 2012 May 15;2(5):e115. doi: 10.1038/tp.2012.46. PMID: 22832959; PMCID: PMC3365265.

Toda, T., Parylak, S.L., Linker, S.B. et al. (2019). The role of adult hippocampal neurogenesis in brain health and disease. Mol Psychiatry 24, 67–87. https://doi.org/10.1038/s41380-018-0036-2

Torgersen S, Lygren S, Oien PA, Skre I, Onstad S, Edvardsen J, Tambs K, Kringlen E (2000). "A twin study of personality disorders". Comprehensive Psychiatry. 41 (6): 416–425. doi:10.1053/comp.2000.16560

Weiner L, Perroud N, Weibel S. (2019). Attention Deficit Hyperactivity Disorder And Borderline Personality Disorder In Adults: A Review Of Their Links And Risks. Neuropsychiatr Dis Treat. 2019 Nov 8;15:3115-3129. doi: 10.2147/NDT.S192871. PMID: 31806978; PMCID: PMC6850677.

Yang Y, Wang JZ. From Structure to Behavior in Basolateral Amygdala-Hippocampus Circuits. (2017). Front Neural Circuits. 2017 Oct 31;11:86. doi: 10.3389/fncir.2017.00086. PMID: 29163066; PMCID: PMC5671506.

Zanarini MC, Reichman CA, Frankenburg FR, Reich DB, Fitzmaurice G. (2010) The course of eating disorders in patients with borderline personality disorder: a 10-year follow-up study. Int J Eat Disord. 2010;43(3):226-32. doi:10.1002/eat.20689

Zanarini MC, Frankenburg FR, Reich DB, et al. (2012) Attainment and stability of sustained symptomatic remission and recovery among patients with borderline personality disorder and Axis II comparison subjects: a 16-year prospective follow-up study. Am J Psychiatry. 2012;169(5):476–483.

Zimmerman, M., (2019). Borderpolar: Patients with Borderline Personality Disorder and Bipolar Disorder. Psychiatric Times, Psychiatric Times Vol 36, Issue 12, Volume 36, Issue 12

,

Tim SerongTANSTAAFL

It’s been a little over a year since our Redflow ZCell battery and Victron Energy inverter/charger kit were installed on our existing 5.94kW solar array. Now that we’re past the Southern Hemisphere spring equinox it seems like an opportune time to review the numbers and try to see exactly how the system has performed over its first full year. For background information on what all the pieces are and what they do, see my earlier post, Go With The Flow.

As we look at the figures for the year, it’s worth keeping in mind what we’re using the battery for, and how we’re doing it. Naturally we’re using it to store PV generated electricity for later use when the sun’s not shining. We are also charging the battery from the grid at certain times so it can be drawn down if necessary during peak times, for example I set up a small overnight charge to ensure there was power for the weekday morning peak, when the sun isn’t really happening yet, but grid power is more than twice as expensive. More recently in the winter months, I experimented with keeping the battery full with scheduled charges during most non-peak times. This involved quite a bit more grid charging, but got us through a couple of three hour grid outages without a hitch during some severe weather in August.

I spent some time going through data from the VRM portal for the last year, and correlating that with current bills from Aurora energy, and then I tried to compare our last year of usage with a battery, to the previous three years of usage without a battery. For reasons that will become apparent later, this turned out to be a massive pain in the ass, so I’m going to start by looking only at what we can see in the VRM portal for the past year.

The VRM portal has three summary views: System Overview, Consumption and Solar. System Overview tells us overall how much total power was pulled from the grid, how much was exported to the grid, how much was produced locally, and how much was consumed by our loads. The Consumption view (which I wish they’d named “Loads”, because I think that would be clearer) gives us the same consumption figure, but tells us how much of that came from the grid, vs. what came from the battery vs. what came from solar. The Solar view tells us how much PV generation went to the grid, how much went to the battery, and how much was used directly. There is some overlap in the figures from these three views, but there are also some interesting discrepancies, notably: the “From Grid” and “To Grid” figures shown under System Overview are higher than what’s shown in the Consumption and Solar views. But, let’s start by looking at the Consumption and Solar views, because those tell us what the system gives us, and what we’re using. I’ll come back after that to the System Overview, which is where things start to get weird and we discover what the system costs to run.

The VRM portal lets you chose any date range you like to get historical figures and bar charts. It also gives you pie charts of the last 24 hours, 7 days, 30 days and 365 days. To make the figures and bar charts match the pie charts, the year we’re analysing starts at 4pm on September 25, 2021 and ends at 4pm on September 25, 2022, because that’s exactly when I took the following screenshots. This means we get a partial September at each end of the bar chart. I’m sorry about that.

Here’s the Consumption view:

Consumption view from VRM portal, 2021-09-25 16:00 – 2022-09-25 16:00

This shows us that in the last 12 months, our loads consumed 10,849kWh of electricity. Of that, 54% (5,848kWh) came from the grid, 23% (2,506kWh) came direct from solar PV and the final 23% (2,494kWh) came from the battery.

From the rough curve of the bar chart we can see that our consumption is lower in the summer months and higher in the winter months. I can’t say for certain, but I have to assume that’s largely due to heating. The low in February was 638kWh (an average of 22.8kWh/day). The high in July was 1,118kWh (average 36kWh/day).

Now let’s look at the Solar view:

Solar view from VRM portal, 2021-09-25 16:00 – 2022-09-25 16:00

In that same time period we generated 5,640kWh with our solar array, of which 44% (2,506kWh) was used directly by our loads, 43% (2,418kWh) went into the battery and 13% (716kWh) was exported to the grid.

Unsurprisingly our generation is significantly higher in summer than in winter. We got 956kWh (average 30kWh/day) in December but only 161kWh (5.3kWh/day) in June. Peak summer figures like that mean we’ll theoretically be able to do without grid power at all during that period once we get a second ZCell (note that we’re still exporting to the grid in December – that’s because we’ve got more generation capacity than storage). The winter figures clearly indicate that there’s no way we can provide anywhere near all our own power at that time of year with our current generation capacity and loads.

Now look closely at the summer months (December, January and February). There should be a nice curve evident there from December to March, but instead January and February form a weird dip. This is because we were without solar generation for three weeks from January 30 – February 11 due to replacing a faulty MPPT. Based on figures from previous years, I suspect we lost 500-600kWh of potential generation in that period.

Another interesting thing is that if we compare “To Battery” on the Solar view (2,418kWh) with “From Battery” on the Consumption view (2,494kWh), we see that our loads consumed 76kWh more from the battery than we actually put into it with solar generation. This discrepancy is due to the fact that in addition to charging the battery from solar, we’ve also been charging it from the grid at certain times, but the amount of power sent to the battery from the grid isn’t broken out explicitly anywhere in the VRM portal.

Now let’s look at the System Overview:

System Overview view from VRM portal, 2021-09-25 16:00 – 2022-09-25 16:00

Here we see the same figures for “Production” (5,640kWh) and “Consumption” (10,849kWh) as were in the Consumption and Solar views, and the bar chart shows the same consumption and generation curves (ignore the blue overlay and line which indicate battery minimum/maximum and average state of charge – that information is largely meaningless at this scale, given we cycle the battery completely every day).

Now look at “To Grid” and “From Grid”. “To Grid” is 754 kWh, i.e. we somehow sent 38kWh more to the grid than came from solar. “From Grid”, at 8,531kWh, is a whopping 2,683kWh more than the 5,848kWh grid power consumed by our loads (i.e. close to half as much again).

So, what’s going on here?

One factor is that we’re charging the battery from the grid at certain times. Initially that was a few hours overnight and a few hours in the afternoon on weekdays, although the afternoon charge is obviously also provided by the solar if the sun is shining. For all of July, August and most of September though I was using a charge schedule to keep the battery full except for peak times and maintenance cycle nights, which meant quite a bit more grid charging overnight than earlier in the year, as well as grid charging most of the day during days with no or minimal sunshine. Grid power sent to the battery isn’t visible in the “From Grid” figure on the Consumption view – that view shows only our loads, i.e. the equipment the system is powering – but it is part of the “From Grid” figure in the System Overview.

Similarly, some of the power we export to the grid is actually exported from the battery, as opposed to being exported from solar generation. That usually only happens during maintenance cycles when our loads aren’t enough to draw the battery down at the desired discharge rate. But again, same thing, that figure is present here on the system overview page as part of “To Grid”, but of course is not part of the “To Grid” figure on the Solar view.

Another factor is that the system itself needs some amount of power to operate. The Victron kit (the MultiPlus II Inverter/Chargers, the Cerbo GX, the MPPT) use some small amount of power themselves. The ZCell battery also requires power to operate its pumps and fans. When the sun is out this power can of course come from solar. When solar power is not available, power to run the system needs to come from some combination of the remaining charge in the battery, and the grid.

On that note, I did a little experiment to see how much power the system uses just to operate. On July 9 (which happened to be a maintenance cycle day), I disabled all scheduled battery charges, and I shut off the DC isolators for the solar PV, so the battery would remain online (pumps and fans running) but empty for all of July 10. The following day I went and checked the figures on the System Overview, which showed we drew 35kWh, but that our consumption was 33kWh. So, together, the battery doing nothing other than running its pumps and fans, plus the Multis doing nothing other than passing grid power through, used 2kWh of power in 24 hours. Over a year, that’s 730kWh. As mentioned above, ordinarily some of that will be sourced from mains and some from solar, but if we look at the total power that came into the system as a whole (5,640kWh from solar + 8,531kWh from the grid = 14,171kWh), 730kWh is just slightly over 5% of that.

The final factor in play is that a certain amount of power is naturally lost due to conversion at various points. The ZCell has a maximum 80% DC-DC stack efficiency, meaning in the absolute best case if you want to get 10kW out of it, you have to put 12.5kW in. In reality you’ll never hit the best case: the lifetime charge and discharge figures the BMS currenly shows for our ZCell are 4,423 and 3,336kWh respectively, which is a bit over 75%. The Multis have a maximum efficiency of 96% when doing their invert/charge dance, so if we grid charge the battery, we lose at least 4% on the way in, and at least 4% on the way out as well, going to and from AC/DC. Again, in reality that loss will be higher than 4% each way, because 96% is the maximum efficiency.

A bunch of the stuff above just doesn’t apply to the previous system with the ABB inverter and no battery. I also don’t have anything like as much detailed data to go on for the old system, which makes comparing performance with the new system fiendishly difficult. The best comparison I’ve been able to come up with so far involves looking at total power input to the system (power from grid plus solar generation), total consumption by loads (i.e. actual locally usable power), and total power exported.

Prior to the Victron gear and Redflow battery installation, I had grid import and export figures from my Aurora Energy bills, and I had total generation figures from the ABB inverter. From this I can synthesise what are hopefully reasonably accurate load consumption figures by adding adding grid input to total PV generation minus grid export.

I had hoped to do this analysis on a quarterly basis to line up with Aurora bills, because then I would also be able to see how seasonal solar generation and usage went up and down. Unfortunately the billing for 2020 and 2021 was totally screwed up by the COVID-19 pandemic, because there were two quarters during which nobody was coming out to read the electricity meter. The bills for those quarters stated estimated usage (i.e. were wrong, especially given they estimated grid export as zero), with subsequent quarters correcting the figures. I have no way to reliably correlate that mess with my PV generation figures, except on an annual basis. Also, using billing periods from pre-battery years, the closest I can get to the September 25 based 2021-2022 year I’m looking at now is billing periods starting and ending in mid-August. But, that’s close enough. We’ve still got four pretty much back-to-back 12 month periods to look at.

YearGrid InSolar InTotal InLoadsExport
2018-20199,0316,68215,71311,8273,886
2019-20209,3246,46815,79212,2553,537
2020-20217,5826,34713,92910,3583,571
2021-20228,5315,64014,17110,849754

One thing of note here is that in the 2018-2019 and 2019-2020 years, our annual consumption was pretty close to 12MWh, whereas in 2020-2021 and 2021-2022 it was closer to 10.5MWh. If I had to guess, I’d say that ~1.5MWh/year drop is due to a couple of pieces of computer equipment that were previously always on, now mostly running in standby mode except when actually needed. A couple of hundred watts constant draw is a fair whack of power over the course of a year. Another thing to note is the big drop in power exported in 2021-2022, because most of our solar generation is now used locally.

The thing that freaked me out when looking at these figures is that in the battery year, while our loads consumed 491kWh more than in the previous non-battery year, we pulled 949kWh more power in from the grid! This is the opposite of what I had expected to see, especially having previously written:

In the eight months the system has been running we’ve generated 4631kWh of electricity and “only” sent 588kWh to the grid, which means we’ve used 87% of what we generated locally – much better than the pre-battery figure of 45%. I suspect we’ve reduced the amount of power we pull from the grid by about 30% too, but I’ll have to wait until we have a full year’s worth of data to be sure.

– by me at the end of Go With The Flow

When I wrote that, I was looking at August 31, 2021 through April 27, 2022, and comparing that to the August 2020 to May 2021 grid power figures from my old Aurora bills. The mistake I must have made back then was to look at “From Grid” on the Consumption view, rather than “From Grid” on the System Overview. I’ve just done this exercise again, and the total grid draw from our Aurora bills from August 2020 to May 2021 is 4,980kWh. “From Grid” on the Consumption view for August 2021 to May 2022 is 3,575kWh, which is about 30% less, but “From Grid” on the System Overview is 4,754kWh, which is only about 5% less. So our loads pulled about 30% less from the grid than the same time the year before, but our system as a whole didn’t.

Now let’s break our ridiculous September-based year down further into months, to see if we can see more detail. I’ve highlighted some interesting periods in bold.

MonthGrid InSolar InTotal InLoadsExport
Sep 21 (part)1531012542136
Oct 216366291,26598855
Nov 214307471,17786697
Dec 212329561,188767176
Jan 226524501,10282274
Feb 2247043090063883
Mar 224985681,06681364
Apr 2260937798677527
May 229102381,1489533
Jun 221,1141611,27510732
Jul 221,1632231,386111811
Aug 229103751,28596664
Sep 22 (part)7543851,13985792
Total8,5315,64014,17110,849754

December is great. We generated about 25% more power than our loads use (956/767=1.25), and our grid input was only about 30% of the total of our loads (232/767=0.30).

January and February show the effects of missing three weeks of potential generation. I mean, just look at December through February 2021-2022 versus the previous three summers.

PV Generation December through January 2018-2022
 2018-20192019-20202020-20212021-2022
December919882767956
January936797818450
February699656711430

June and July are terrible. They’re our highest load months, with the lowest solar generation and we pulled 3-4% more power from the grid than our loads actually consumed. I’m going to attribute the latter largely to grid charging the battery.

If I dig a couple of interesting figures out for June and July I see “To Battery” on the Solar view shows 205kWh, and “From Battery” on the Consumption view shows 558kWh. Total consumption in that period was 2,191kWh, with the total “From Grid” reported in System Overview of 2,277kWh. Let’s mess with that a bit.

Bearing in mind the efficiency numbers mentioned earlier, if 205kWh went to the battery from PV, that means no more than 154kWh of what we got out of the battery was from PV generation (remember: real world DC-DC stack efficiency of about 75%). The remaining 404kWh out of the battery is power that went into it from the grid. And that means at least 538kWh in (404/0.75). Note that total from grid for these two months was 86kWh more than the 2,191kWh used by our loads. If I hadn’t been keeping the battery topped up from the grid, I’d’ve saved at least 134kWh of grid power, which would have brought our grid input figure back down below our consumption figure. Note also that this number will actually be higher in reality because I haven’t factored in AC/DC conversion losses from the Multis.

Now let’s look at some costs. When I started trying to compare the new system to the previous system, I went in thinking to look at in in terms of total power input to the system, total consumption by loads, and total power exported. There’s one piece missing there, so let’s add another couple of columns to an earlier table:

YearGrid InSolar InTotal InLoadsExportTotal Outwhat?
2021-20228,5315,64014,17110,84975411,6032,568

The total usable output of the system was 11,603kWh for 14,171kWh input. The difference between these two figures – 2,568kWh, or about 18% – went somewhere else. Per my earlier experiment, 5% is power that went to actually operate the system components, including the battery. That means about 13% of the power input to the system over the course of the year must have gone to some combination of charge/discharge and AC/DC conversion (in)efficiencies. We can consider this the energy cost of the system. To have the ability to time-shift expensive peak grid electricity, and to run the house without the grid if the sun is out, or from the battery when it has charge, costs us 18% of the total available energy input.

Grid power has energy costs too, but we’re not usually aware of this because it happens somewhere else. I haven’t yet found Tasmanian figures, but this 2021 Transmission Annual Planning Report PDF from Powerlink in Queensland has historical figures showing that about 7% of generation there went to auxiliaries, i.e. fans and pumps and things running at the power stations. And according to the Australian Energy Market Operator (AEMO), 10% of grid power generated is lost during transmission and distribution. Stanwell (a power company in Queensland) have a neat explainer of all this on their What’s Watt site.

Finally, speaking of expensive grid electricity, let’s look at how much we paid Aurora Energy over the past four years for our power. The bills are broken out into different tariffs, for which you’re charged different amounts per kilowatt hour and then there’s an additional daily supply charge, and also credits for power exported. We can simplify that by just taking the total dollar value of all the power bills and dividing that by the total power drawn from the grid to arrive at an effective cost per kilowatt hour for the entire year. Here it is:

YearFrom GridTotal BillCost/kWh
2018-20199,031$2,278.33$0.25
2019-20209,324$2,384.79$0.26
2020-20217,582$1,921.77$0.25
2021-20228,531$1,731.40$0.20

So, the combination of the battery plus the switch from Flat Rate to Peak & Off-Peak billing has reduced the cost of our grid power by about 20%. I call that a win.

Going forwards it will be interesting to see how the next twelve months go, and, in particular, what we can do to reduce our power consumption. A significant portion of our power is used by a bunch of always-on computer equipment. Some of that I need for my work, and some of that provides internet access, file storage and email for us personally. Altogether, according to the UPSes, this kit pulls 200-250 watts continuously, but will pull more than that during the day when it’s being used interactively. If we call it 250W continuous, that’s a minimum of 6kWh/day, which is 2,190kWh/year, or about 20% of the 2021-2022 consumption. Some of that equipment should be replaced with newer, more power efficient kit. Some of it could possibly even be turned off or put into standby mode some of the time.

We still need to get a heat pump to replace the 2400W panel heater in our bedroom. That should save a huge amount of power in winter. We’re also slowly working our way through the house installing excellent double glazed windows from Elite Double Glazing, which will save on power for heating and cooling year round.

And of course, we still need to get that second ZCell.

,

Lev LafayetteCompiling Your Python

It still generates a little bit of surprise to discover that there are people who use Python on a daily basis that are apparently quite unfamiliar with compiling said code. Or perhaps not; it is, after all, the world's most popular programming language, it has a syntax that's cleaner than many older languages, it has an enormous collection of extensions, and so forth. As a result, there are many people who use Python, but perhaps not so many who have the inquisitiveness and courage, to dive a little deeper. This short article is a deeper dive to understand a little more about the language.

The first common (novice) mistake is that Python is an interpreted language and can't be compiled. It most certainly can and is "compiled", but not in the same way that a compiled language (e.g., C/C++, Fortran, Pascal) is. If this sounds confusing one needs to dig a little into the architecture.

A Python program is compiled before being interpreted, but this step is hidden at the surface level. When Python is executed it generates byte code. This byte code is transformed and interpreted by the Python Virtual Machine which then converts the byte code to binary machine code that the computer processor can output.

A top-level Python source file can be compiled in the following manner:


$ python -m py_compile helloworld.py

Or, with multiple files:


$ python -m py_compile helloworld.py hellosystem.py hellogalaxy.py ...

Running this will generate a __pycache__ that will contain the byte-code (e.g., helloworld.cpython-38.pyc) version of the program. This binary byte-code can be directly invoked (e.g., python3 helloworld.cpython-38.pyc).

The distinction between byte code interpreted by the PVM and machine code is very important, as sometimes people believe (armed with this new information) that because Python can be compiled that this will lead to much faster execution. This is incorrect. A compiled Python script does not run any faster than a non-compiled script. So why would one want to do this?

The short answer is that a program doesn't just run, it must also load its environment and including any other programs, modules, packages, etc that have been imported. If the runtime of a program is quite long then the advantage of compiling a Python program is relatively less, and conversely, a Python program with a short runtime or one that imports a number of additional programs is going to have a more significant advantage. In any case, there will be some performance improvement because all Python programs will be converted to byte code and interpreted by the PVM anyway. The example helloworld.py has an improvement of over 300% in real time, for example. This is not bound by simplicity or length, by the way. For example, a complex mathematical Python program can take a very long time to load, but runtime execution is very quick. Testing, as always, is the best solution.

There are two other advantages to using compiled Python. The first is that the byte code provides some protection against unwanted changes in a shared environment, for example, a careless contributor who in their passion to fix what looks like a bug makes a modification to the source code and breaks the program. Of course, a version control system should be used but even then an additional layer of protection is often worthwhile. The second advantage is that compilation often will result in a significantly smaller file and when coupled with the faster load times, this is particularly useful for websites, embedded environments, and especially when using Python for network programming. Seriously, in such an environment when it's waiting for a trigger it is critical for a fast response; this can be achieved by having the modules like socket, stmplib, base64, etc already loaded.

All silver linings are attached to clouds, however, and compiling Python is no different. The byte code is designed for the particular system architecture it was compiled in and therefore is not exactly transportable to a new system: "different machine, different Python", as the saying goes. Thus distribution will, at the very least, require distributing all the relevant .py files for implementation.

There are, being computing, many further elaborations that one could engage in. For example, I have not discussed the deeper optimisation options for compilation and the pitfalls that could result. Nor have I raised the details of the interpreter and the evaluation loop, or the distinction between implementations at that level (e.g., CPython, Jython, IronPython, PyPy, etc). Finally, perhaps in a simpler direction, I have not discussed the use of the compile() function which converts s specified source as a code object to be executed, such as by eval() or exec(). These will be discussed at another time. In the meantime, if performance matters, you probably should compile your Python code.

,

Tim RileyOpen source status update, August 2022

August’s OSS work landed one of the last big Hanami features, saw another Hanami release out the door, began some thinking about memory usage, and kicked off a fun little personal initiative. Let’s dive in!

Conditional slice loading in Hanami

At the beginning of the month I merged support for conditional slice loading in Hanami. I’d wanted this feature for a long time, and in fact I’d hacked in workarounds to achieve the same more than 2 years ago, so I was very pleased to finally get this done, and for the implementation work to be as smooth as it was.

The feature provides a new config.slices setting on your app class, which you can configure like so:

module MyApp
  class App < Hanami::App
    config.slices = %w[admin]
  end
end

For an app consisting of both Admin and Main slices and for the config above, when the app is booted, only the Admin slice will be loaded:

require "hanami/prepare"

Hanami.app.slices.keys # => [:admin]

Admin::Slice # exists, as expected
Main         # raises NameError, since it was never loaded

As we see from Main above, slices absent from this list will not have their namespace defined, nor their slice class loaded, nor any of their Ruby source files. Within that Ruby process, they effectively do not exist.

Specifying slices to load can be very helpful to improve boot time and minimize memory usage for specific deployed workloads of your app.

Imagine you have a subset of background jobs that run via a dedicated job runner, but whose logic is otherwise unneeded for the rest of your app to function. In this case, you could organize those jobs into their own slice, and then load only that slice for the job runner’s process. This arrangement would see the job runner boot as quickly as possible (no extraneous code to load) as well as save all the memory otherwise needed by all those classes. You could also do the invserse for your main deployed process: specify all slices except this jobs slice, and you gain savings there too.

Organising code into slices to promote operational efficiency like this also gives you the benefit of greater clarity in the separation of responsibilities between those slices: when a single slice of code is loaded and the rest of your app is made to disappear, that will quickly surface any insidious dependencies from that slice to the rest of your code (they’ll be raised as exceptions!). Cleaning these up will help ensure your slices remain useful as abstractions for reasoning about and maintaining your app.

To make it easy to tune the list of slices to load, I also introduced a new HANAMI_SLICES env var that sets this config without you having to write code inside your app class. In this way, you could use them in your Procfile or other similar deployment code:

web: HANAMI_SLICES=main,admin bundle exec puma -C config/puma.rb
feed_worker: HANAMI_SLICES=feed bundle exec rake jobs:work

This effort was also another example of why I’m so happy to be working alongside the Hanami core team. After initially proposing a more complex arrangement including separate lists for including or excluding slices, Luca jumped in and help me dial this back to the much simpler arrangement of the single list only. For an Hanami release in which we’re going to be introducing so many new ideas, the more we can keep simple around them, the better, and I’m glad to have people who can remind me of this.

Fixed how slice config is applied to component classes

Our action and view integration code relies on their classes detecting when they’re defined inside a slice’s namespace, then applying relevant config from the slice to their own class-level config object. It turned out our code for doing this broke a little when we adjusted our default class hierarchies. Thanks to some of our wonderful early adopters, we picked this up quickly and I fixed it. Now things just work like you expect however you choose to configure your action classes, whether through the app-level config.actions object, or by directly updating config in a base action class.

In doing this work, I became convinced we need an API on dry-configurable to determine whether any config value has been assigned or mutated by the user, since it would help so much in reliably detecting whether or not we should ignore config values at particular levels. For now, we could work around it, but I hope to bring this to dry-configurable at some point in the future.

Released Hanami 2.0.0.beta2

Another month passed, so it was time for another release! With my European colleagues mostly enjoying some breaks over their summer, I hunkered down in chilly Canberra and took care of the 2.0.0.beta2 release. Along with the improvements above, this release also included slice and action generators (hanami generate slice and hanami generate action, thank you Luca!), plus a very handle CLI middlewares inspector (thank you Marc!):

$ hanami middlewares

/    Dry::Monitor::Rack::Middleware (instance)
/    Rack::Session::Cookie

The list of things to do over the beta phase is getting smaller. I don’t expect we’ll need too many more of these releases!

Created memory usage benchmarks for dry-configurable

As the final 2.0 release gets closer, we’ve been doing various performance tests just to make sure the house is in order. One thing we discovered is that Hanami::Action is not as memory efficient as we’d like it to be. One of the biggest opportunities to improve this looked to be in dry-configurable, since that’s what is used to manage the per-class action configuration.

I suspected any effort here would turn out to be involved (and no surprise, it turned out to be involved 😆), so I thought it would be useful as a first step to establish a memory benchmark to revisit over the course of any work. This was also a great way to get my head in this space, which turned out to take over most of my September (but more on that next month).

Quietly relaunched Decaf Sucks

Decaf Sucks was once a thriving little independent online café review community, with its own web site (starting from humble beginnings as a Rails Rumble entry in 2009) and even native iOS app (two iterations, in fact).

I was immensitely proud of what Decaf Sucks became, and for the collaboration with Max Wheeler in building it.

Unfortunately, as various internet APIs changed, the site atrophied, eventually became disfunctional, and we had to take it down. I still have the database, however, and I want to bring it back!

This time around, my plan is to do it as a fully open source Hanami 2 example application. Max is even on board to bring back all the UI goodness. For now, you can follow along with the early steps on GitHub. Right now the app is little more than the basic Hanami skeleton with added database integration and a CI setup (Hello Buildkite!), but I plan to grow it bit by bit. Perhaps I’ll try to have something small that I can share with each of these monthly OSS updates.

After Hanami 2 ships, hopefully this will serve as a useful resource for people wanting to see how it plays out in a real working app. And beyond that, I look forward to it serving once again as a place for me to commemorate my coffee travels!

,

Tim SerongAn S3 Storage Experiment

My team at SUSE is working on a new S3-compatible storage solution for Kubernetes, based on Ceph’s RADOS Gateway (RGW), except without any of the RADOS bits. The idea is that you can deploy our s3gw container on top of Longhorn (which provides the underlying replicated storage), and all this is running in your Kubernetes cluster, along with your applications which thus have convenient access to a local S3-compatible object store.

We’ve done this by adding a new storage backend to RGW. The approach we’ve taken is to use SQLite for metadata, with object data stored as files in a regular filesystem. This works quite neatly in a Kubernetes cluster with Longhorn, because Longhorn can provide a persistent volume (think: an ext4 filesystem), on which s3gw can store its SQLite database and object data files. If you’d like to kick the tyres, check out Giuseppe’s deployment tutorial for the 0.2.0 release, but bear in mind that as I’m writing this we’re all the way up to 0.4.0 so some details may have changed.

While s3gw on Longhorn on Kubernetes remains our primary focus for this project, the fact that this thing only needs a filesystem for backing storage means it can be run on top of just about anything. Given “just about anything” includes an old school two node Pacemaker cluster with DRBD for replicated storage, why not give that a try? I kinda like the idea of a good solid highly available S3-compatible storage solution that you could shove into the bottom of a rack somewhere without too much difficulty.

It’s probably eight years since I last deployed Pacemaker and DRBD, so to refresh my memory I ran with SUSE’s latest Highly Available NFS Storage with DRBD and Pacemaker document, but skipped all the NFS bits. That gives a filesystem mounted on one node, which will fail over to the other node if something breaks. On top of that, we need to run the s3gw container, the s3gw-ui container, an nginx HTTPS reverse proxy to smoosh those two together, and a virtual/floating IP, so the whole lot is accessible to the outside world.

Here’s the interesting parts of my Pacemaker configuration:

# crm configure show
[...]
primitive drbd_s3 ocf:linbit:drbd \
        params drbd_resource=s3 drbdconf="/etc/drbd.conf" \
        op monitor interval=29s role=Master \
        op monitor interval=31s role=Slave
primitive fs_s3 Filesystem \
        params device="/dev/drbd0" directory="/data" fstype=ext4 \
        meta target-role=Started \
        op start timeout=60s interval=0 \
        op stop timeout=60s interval=0 \
        op monitor interval=20s timeout=40s
primitive https nginx \
        op start timeout=40s interval=0 \
        op stop timeout=60s interval=0 \
        op monitor timeout=30s interval=10s \
        op monitor timeout=30s interval=30s \
        op monitor timeout=60s interval=20s
primitive s3-ip IPaddr2 \
        params ip=192.168.100.50 \
        op monitor interval=10 timeout=20
primitive s3gw podman \
        params image="ghcr.io/aquarist-labs/s3gw:latest" run_opts="-p 7480:7480 -v/data:/data" \
        op start interval=0 timeout=90s \
        op stop interval=0 timeout=90s \
        op monitor interval=30s timeout=30s
primitive s3gw-ui podman \
        params image="ghcr.io/aquarist-labs/s3gw-ui:latest" run_opts="-p 8080:8080 -e RGW_SERVICE_URL=https://s3gw.sleha.test" \
        op start interval=0 timeout=90s \
        op stop interval=0 timeout=90s \
        op monitor interval=30s timeout=30s
group g-s3 fs_s3 s3gw s3gw-ui https s3-ip
ms ms-drbd_s3 drbd_s3 \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
colocation col-s3_on_drbd inf: g-s3 ms-drbd_s3:Promoted
order o-drbd_before_fs Mandatory: ms-drbd_s3:promote g-s3:start
[...]

The g-s3 group ensures that the ext4 filesystem (fs_s3), s3gw container (s3gw), s3gw-ui container (s3gw-ui), nginx instance (https) and virtual IP (s3-ip) all run on the same node, and start one after another. The colocation and ordering constraints ensure that g-s3 runs on whichever node is currently the DRBD (ms-drbd_s3) primary.

The important pieces of glue here are:

  • The fs_s3 resource mounts /dev/drbd0 on /data
  • The s3gw resource passes -p 7480:7480 -v/data:/data to podman, so the container can write to /data on the host, and the S3 service is accessible via HTTP on port 7480.
  • The s3gw-ui resource passes -p 8080:8080 -e RGW_SERVICE_URL=https://s3gw.sleha.test to podman, so the UI is accessible via HTTP on port 8080, and it expects the S3 service to be externally available via https://s3gw.sleha.test.
  • nginx is configured to reverse proxy https://s3gw.sleha.test to http://localhost:7480, and https://s3gw-ui.sleha.test to http://localhost:8080.
  • I’ve got an entry in /etc/hosts to point s3gw.sleha.test and s3gw-ui.sleha.test at the virtual IP (192.168.100.50).
  • I’m using self-signed certificates (openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout cert.key -out cert.pem) for s3gw and s3gw-ui, so I had to go visit both https://s3gw.sleha.test and https://s3gw-ui.sleha.test in my browser and accept the SSL certificate before the UI would work.
  • The DRBD config, nginx config and SSL certificates and keys need to be present on all nodes. I used csync2 for this.

Here’s my /etc/nginx/nginx.conf. I’m not entirely convinced I’ve got everything 100% right here, but it seems to work (this is, incredibly, my first time doing anything with nginx, and my first time dealing with CORS):

worker_processes  1;

events {
    worker_connections  1024;
    use epoll;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        return       301 https://$host$request_uri; 
    }

    server {
        listen       443 ssl;
        server_name  s3gw.sleha.test;

        access_log /var/log/nginx/s3gw.access.log;

        location / {
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;

            add_header Access-Control-Allow-Origin 'https://s3gw-ui.sleha.test';
            add_header Access-Control-Allow-Methods 'GET,HEAD,PUT,POST,DELETE';
            add_header Access-Control-Allow-Headers '*';
            add_header 'Access-Control-Allow-Credentials' 'true';

            if ($request_method = 'OPTIONS') {
                add_header Access-Control-Allow-Origin 'https://s3gw-ui.sleha.test';
                add_header Access-Control-Allow-Methods 'GET,HEAD,PUT,POST,DELETE';
                add_header Access-Control-Allow-Headers '*';
                add_header 'Access-Control-Allow-Credentials' 'true';
                add_header 'Content-Type' 'text/plain charset=UTF-8';
                add_header 'Content-Length' 0;
                return 204;
            }

            proxy_pass          http://localhost:7480;
            proxy_read_timeout  90;
            proxy_redirect      http://localhost:7480 https://s3gw.sleha.test;
        }

        ssl_certificate      cert.pem;
        ssl_certificate_key  cert.key;
        ssl_protocols        TLSv1.2;
        ssl_session_cache    shared:SSL:1m;
        ssl_session_timeout  5m;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;
    }

    server {
        listen       443 ssl;
        server_name  s3gw-ui.sleha.test;

        access_log /var/log/nginx/s3gw-ui.access.log;

        location / {
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;

            proxy_pass          http://localhost:8080;
            proxy_read_timeout  90;

            proxy_redirect      http://localhost:8080 https://s3gw-ui.sleha.test;
        }

        ssl_certificate      cert-ui.pem;
        ssl_certificate_key  cert-ui.key;
        ssl_protocols        TLSv1.2;
        ssl_session_cache    shared:SSL:1m;
        ssl_session_timeout  5m;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;
    }
}

A couple of important points about Pacemaker’s support for running containers with podman:

So what was the end result? TL;DR: It pretty much All Just WorkedTM, which is exactly what you’d hope for when running a new application on a mature HA stack. I can use s3cmd to mess around with the S3 service, and use my web browser to play with the UI. Failover is nice and quick (think: a few seconds) if I kill a node. For the sake of convenience I did this experiment on a couple of VMs using the external/libvirt STONITH plugin, but I don’t expect a real deployment to be hugely different in behaviour. Also, I’d forgotten how good Pacemaker is at highlighting poorly behaved applications – prior to this experiment the s3gw-ui container didn’t stop well, but we weren’t aware of that until I tried a manual failover which took too long and resulted in an unexpected STONITH due to a stop timeout. Moritz has since fixed that.

One thing I tripped over when doing this deployment was the correct values to use for the access_key and secret_key of the default user when talking to the S3 service. These are actually settable for the s3gw container via the RGW_DEFAULT_USER_ACCESS_KEY and RGW_DEFAULT_USER_SECRET_KEY environment variables, but if left unset, they default to “test” and “test” respectively. The interesting bits of my s3cmd.cfg are thus:

access_key = test
secret_key = test
host_base = https://s3gw.sleha.test/
host_bucket = htts://s3gw.sleha.test/%(bucket)

In retrospect I probably should have added -e RGW_DEFAULT_USER_ACCESS_KEY=tserong -e RGW_DEFAULT_USER_SECRET_KEY=do_not_tell_anyone_this_is_your_password to the run_opts parameter of the s3gw resource in the Pacemaker config.

,

Tim RileyOpen source status update, May–July 2022

Hi there friends, it’s certainly been a while, and a lot has happened across May, June and July: I left my job, took some time off, and started a new job. I also managed to get a good deal of open source work done, so let’s take a look at that!

Released Hanami 2.0.0.alpha8

Since we’d skipped a month in our releases, I helped get Hanami 2.0.0.alpha8 out the door in May. The biggest change here was that we’d finished relocating the action and view integration code into the hanami gem itself, wrapped up in distinct “application� classes, like Hanami::Application::Action. In the end, this particular naming scheme turned out to be somewhat short lived! Read on for more :)

Resurrected work using dry-effects within hanami-view

As part of an effort to make it easy to use our conventional view “helpers� in all parts of our view layer, I resurrected my work from September 2020(!) on using dry-effects within hanami-view. The idea here was to achieve two things:

  1. To ensure we keep only a single context object for the entire view rendering, allowing its state to be preserved and accessed by all view components (i.e. allowing both templates, partials and parts all to access the very same context object)
  2. To enable access to the current template/partial’s #locals from within the context, which might help make our helpers feel a little more streamlined through implicit access to those locals

I got both of those working (here’s my work in progress), but I discovered the performance had worsened due to the cost of using an effect to access the locals. I took a few extra passes at this, reducing the number of effects to one, and memoziing it, leaving us with improved performance over the main branch, but with a slightly different stance: the single effect is for accessing the context object only, so any helpers, instead of expecting access to locals, will instead only have access to that context. The job from here will be to make sure that the context object we build for Hanami’s views has everything we need for an ergonomic experience working with our helpers. I’m feeling positive about the direction here, but it’ll be a little while before I get back to it. Read on for more on this (again!).

Unified application and slice

The biggest thing I did over this period was to unify Hanami’s Application and Slice. This one took some doing, and I was glad that I had a solid stretch of time to work on it between jobs.

I already wrote about this back in April’s update, noting that I’d settled on the approach of having a composed slice inside the Hanami::Application class to providing slice-like functionality at the application level. This was the approach I continued with, and as I went, I was able to move more and more functionality out of Hanami::Application and into Hanami::Slice, with that composed “application slice� being the thing that preserved the existing application behaviour. At some point, a pattern emerged: the application is a slice, and we could achieve everything we wanted (and more) by turning class Hanami::Application into class Hanami::Application < Hanami::Slice.

Turning the application into a slice sublcass is indeed how I finished the work, and I’m extremely pleased with how it turned out. It’s made slices so much more powerful. Now, each slice can have its own config, its own dedicated settings and routes, can be run on its own as a Rack application, and can even have its own set of child slices.

As a user of Hanami you won’t be required to use all of this per-slice power features, but they’ll be there if or when you want them. This is a great example of progressive disclosure, a principle I follow as much as possible when designing Hanami’s features: a user should be able to work with Hanami in a simple, straightforward way, and then as their needs grow, they can then find additional capabilities waiting to serve them.

Let’s explore this with a concrete example. If you’re building a simple Hanami app, you can start with a single top-level config/settings.rb that defines all of the app’s own settings. This settings object is made available as a "settings" component registration in both the app as well as all its slices. As the app grows and you add a slice or two, you start to add more slice-specific settings to this component. At this point you start to feel a little uncomfortable that settings specific to SliceA are also available inside SliceB and elsewhere. So you wonder, could you go into slices/slice_a/ and drop a dedicated config/settings.rb there? The answer to that is now yes! Create a config/settings.rb inside any slice directory and it will now become a dedicated settings component for that slice alone. This isn’t a detail you had to burden yourself with in order to get started, but it was ready for you when you needed it.

Another big benefit of this code reorganisation is that the particular responsibilities of Hanami::Application are much clearer: its job is to provide the single entrypoint to the app and coordinate the overall boot process; everything else comes as part of it also being a slice. This distinction is made clear through the number of public methods that exist across the two classes: Application now has only 2 distinct public methods, whereas Slice currently brings 27.

There’s plenty more detail over in the pull request: go check it out!

The work here also led to changes across the ecosystem:

This is one the reasons I’m excited about Hanami’s use of the dry-rb gems: it’s pushing them in directions no one has had to take them before. The result is not only the streamlined experience we want for Hanami, but also vastly more powerful underpinnings.

Devised a slimmed down core app structure

While I had my head down working on internal changes like the above, Luca had been thinking about Hanami 2 adoption and the first run user experience. As we had opted for a slices-only approach for the duration of our alpha releases, it meant a fairly bulky overall app structure: every slice came with multiple deeply nested files. This might be overwhelming to new users, as well as feeling like overkill for apps that are intended to start small and stay small.

To this end, we agreed upon a stripped back starter structure. Here’s how it looks at its core (ignoring tests and other general Ruby files):

├── app/
│   ├── action.rb
│   └── actions/
├── config/
│   ├── app.rb
│   ├── routes.rb
│   └── settings.rb
├── config.ru
└── lib/
    ├── my_app/
    │   └── types.rb
    └── tasks/

That’s it! Much more lightweight. This approach takes advantage of the Hanami app itself becoming a fully-featured slice, with app/ now as its source directory.

In fact, I took this opportunity to unify the code loading rules for both the app and slices, which makes for a much more intuitive experience. You can now drop any ruby source file into app/ or a slices/[slice_name]/ slice dir and it will be loaded in the same way: starting at the root of each directory, classes defined therein are expected to inhabit the namespace that the app or slice represents, so app/some_class.rb would be MyApp::SomeClass and slices/my_slice/some_class would be MySlice::SomeClass. Hat tip to me of September 2021 for implementing the dry-system namespaces feature that enabled this! 😜

(Yet another little dry-system tweak came out of preparing this too, with Component#file_name now exposed for auto-registration rules).

This new initial structure for starter Hanami 2.0 apps is another example of progressive disclosure in our design. You can start with a simple all-in-one approach, everything inside an app/ directory, and then as various distinct concerns present themselves, you can extract them into dedicated slices as required.

Along with this, some of our names have become shorter! Yes, “application� has become “app� (and Hanami::Application has become Hanami::App, and so on). These shorter names are easier to type, as well as more reflective of the words we tend to use when verbally describing these structures.

We also tweaked our actions and views integration code so that it is automatically available when you inherit directly from Hanami::Action, so it will no longer be necessary to have the verbose Hanami::Application::Action as the superclass for the app’s actions. We also ditched that namespace for both routes and settings too, so now you can just inherit from Hanami::Settings and the like.

Devised a slimmed down release strategy

Any of you following my updates would know by now that the Hanami 2.0 release has been a long time coming. We have ambitious goals, we’re doing our best, and everything is slowly coming together. But as hard as it might’ve been for folks who’re waiting, it’s been doubly so for us, feeling the weight of both the work along with everyone’s expectations.

So to make sure we can focus our efforts and get something out the door sooner rather than later, we decided to stagger our 2.0 release. We’ll start off with an initial 2.0 release centred around hanami, hanami-cli, hanami-controller, and hanami-router (enough to write some very useful API applications, for example), then follow up with a “full stack� 2.1 release including database persistence, views, helpers, assets and everything else.

I’m already feeling empowered by this strategy: 2.0 feels actually achievable now! And all of the other release-related work like updated docs and a migration guide will become correspondingly easier too.

Released Hanami 2.0.0.beta1!

With greater release clarity as well as all the above improvements under our belt, it was time to usher in a new phase of Hanami 2.0 development, so we released 2.0.0.beta1 in July! This new version suffix represents just how close we feel we are to our final vision for 2.0. This is an exciting moment!

And a bunch more

This update is getting rather long, so let me list a bunch of other Hanami improvements I managed to get done:

Outside my Hanami development, a new job and a new computer meant I also took the change to reboot my dotfiles, which are now powered by chezmoi. I can’t speak highly enough of chezmoi, it’s an extremely powerful tool and I’m loving the flexibility it affords!

That’s it from me for now. I’ll come back to you all in another month!

,

Lev LafayettePsychic Vampires from Without and Within

The past few days I've been quite influenced by a short essay by Brianna Wiest, which starts with quite a kicker: "It is the hardest thing you will ever have to do, and it will also be the most important: stop giving your love to those who aren’t ready to love you. Stop having hard conversations with people who don't want to change." Ultimately, this is arguing for equality in all relationships. Equality is not a matter of capability or social position. Everyone is unequal in some regard; some people are stronger, some people are smarter, some people know more, some people are richer or have higher status, etc. Individual differences and capabilities are inevitable. Social differences, especially systemic differences (wealth, power) are a little out of our control, barring social change through effective collective effort.

But between people, the one thing that we can be equal in our relationships is concern, care, and effort relative to personal capacity. In other words, if you make a consistent effort to help a person when they are in need, and they are conspicuously absent when you need help, then it is probably best that you redirect your attention elsewhere. As charming as they may be (and they often are when they are at their best) one needs to develop protection against psychic vampires because ultimately they will hurt you.

Psychic vampires exist along a continuum of conscious to unconscious action. On the first extreme, we're dealing with people who are almost certainly are narcissistic [psycho, socio]paths, conscious manipulators who are empowered by the emotional energy they drain from their dependent flock. Obviously, such people are to be avoided at all costs and best left to the professionals to deal with. On the other extreme are those who due to their own impulsivity or lack of security, is a lot needier and have more affective instability. They may even be aware of their vampiric behaviour and feel terribly guilty about it. Unlike the pathological, if one is willing, able, and patient, then these vampiric souls can become a better version of themselves. It will take time and effort because they are, in effect, learning to establish new and unfamiliar pathways in their own brain. Some vampires you want to dispel; others you wish to turn.

Letting go does not just involve vampiric people but also vampiric activities and attachments in one's own life. Most psychic vampires are not like that all the time (indeed, some can be utterly amazing people when they're not being draining), and all of us can be a little bit vampiric at times. Identifying our own vampiric behaviours towards others and toward ourselves, is essential. Are we wasting other people's time and draining their energy? Are we wasting our own time and our energy? When one makes an assessment of the time and activities that we engage in that do not help ourselves and others, when supposedly so-called fun and harmless distractions take up excess time and emotional attachment, then we're in the realm of vampirism. We're literally sucking the life from others and from ourselves, retarding our own personal development and empathy. It is little wonder to discover that clarity in one's own self-concept and associated activities also enhance empathy towards others. This brings us back to the original essay; how do you think empathic people treat others?

,

Ian BrownHigh Velocity Migrations with GCVE and HCX

What is HCX? VMware HCX is an application mobility platform designed for simplifying application migration, workload rebalancing and business continuity across datacenters and clouds. VMware HCX was formerly known as Hybrid Cloud Extension and NSX Hybrid Connect. GCVE HCX GCVE deploys the Enterprise version of HCX as part of the cost of the solution. HCX Enterprise has the following benefits: Hybrid Interconnect WAN Optimisation Bulk Migration, Live Migration and HCX Replication Assisted vMotion Cloud to cloud migration Disaster Protection KVM & Hyper-V to vSphere migrations Traffic Engineering Mobility Groups Mobility Optimised Networking Changeover scheduling Definitions Cold Migration

,

Ian BrownInfrastructure as Code with Terraform in GCVE

We have seen a lot of Google Cloud VMware Engine over the last few months and for the entire time we have used click-ops to provision new infrastructure, networks and VM’s. Now we are going to the next level and we will be using Terraform to manage our infrastructure as code so that it is version controlled and predictable. Installing Terraform The first part of getting this working is installing Terraform on your local machine.

,

Tim SerongHack Week 21: Keeping the Battery Full

As described in some detail in my last post, we have a single 10kWh Redflow ZCell zinc bromine flow battery hooked up to our solar PV via Victron inverter/chargers. This gives us the ability to:

  • Store almost all the excess energy we generate locally for later use.
  • When the sun isn’t shining, grid charge the battery at off-peak times then draw it down at peak times to save on our electricity bill (peak grid power is slightly more than twice as expensive as off-peak grid power).
  • Opportunistically survive grid outages, provided they don’t happen at the wrong time (i.e. when the sun is down and the battery is at 0% state of charge).

By their nature, ZCell flow batteries needs to undergo a maintenance cycle at least every three days, where they are discharged completely for a few hours. That’s why the last point above reads “opportunistically survive grid outages”. With a single ZCell, we can’t use the “minimum state of charge” feature of the Victron kit to always keep some charge in the battery in case of outages, because doing so conflicts with the ZCell maintenance cycles. Once we eventually get a second battery, this problem will go away because the maintenance cycles automatically interleave. In the meantime though, as my project for Hack Week 21, I decided to see if I could somehow automate the Victron scheduled charge configuration based on the ZCell maintenance cycle timing, to always keep the battery as full as possible for as long as possible.

There are three goals somewhat in tension with each other here:

  • Keep the battery full, except during maintenance cycles.
  • Don’t let the battery get too full immediately before a maintenance cycle, lest the discharge take too long and maintenance still be active the following morning.
  • Don’t schedule charges during peak electricity times (we still want to draw the battery down then, to avoid using the expensive gold plated electrons the power company sends down the wire between 07:00-10:00 and 16:00-21:00).

Here’s the solution I came up with:

  • On non-maintenance cycle days, set two no-limit scheduled charges, one from 10:00 for 6 hours, the other from 21:00 for 10 hours. That means the battery will be charged from the grid and/or the sun continuously, except for peak electricity times, when it will be drawn down. Our loads aren’t high enough to completely deplete the battery during peak times, so there will always be some juice in case of a grid outage on non-maintenance cycle days.
  • On maintenance cycle days, set a 50% limit scheduled charge from 13:00 for 3 hours, so the battery won’t be too full before that evening’s maintenance cycle, which kicks in at sunset. The day after a maintenance cycle, set a no limit scheduled charge from 03:00 for 4 hours. At our site, maintenance has almost always finished before 03:00, so there’s no conflict here, and we still have time to get some charge into the battery to handle the next morning’s peak.

Now, how to automate that?

The ZCell Battery Management System (BMS) has a REST API which we can query to find out useful information about the battery. Unfortunately it won’t actually tell us for certain whether maintenance will be run on any given day, but we can get the maintenance time limit, and subtract from that the amount of time that’s passed since the last maintenance cycle. If the resultant figure is less than one day, we know that maintenance will happen today. It is possible for maintenance to happen at other times, e.g. I can force maintenance manually, and also it can happen more often than every three days if you mess with the allowed days setting in the BMS, so this solution arguably isn’t perfect, but I think it’s good enough under the circumstances, at least at our site.

The Victron Cerbo GX (the little box that controls everything) runs Linux, and you can easily get root on it, so it’s possible to write scripts that run locally there. Here’s what I ended up with:

One important point about installing things on the Cerbo GX, is that the root partition is overwritten during firmware updates, but there’s a separate data partition which is preserved. The root user’s home directory is symlinked to /data/home/root, so my script lives at /data/home/root/sched.py to ensure it remains present. Then we need to get it into /etc/crontab, which doesn’t survive firmware updates. This is done by adding a /data/rc.local script which the Cerbo GX runs on boot:

After a few days of testing and observation, I can confirm that it all works perfectly! At least, at our site, right now, with our current loads and daylight ours. The whole thing will want revisiting (or probably just turning off) as we get into summer, when we’ll be able to rely on significantly more sunlight to keep the battery full than we get now. I may well just go back to a single 03:00-for-four-hours grid charge then, once the days are nice and long. See how we go…

,

Tim RileyJoining Buildkite, and sticking with Ruby

Last week I finished up at Culture Amp, and I’m excited to announce that I’ll be joining Buildkite as an engineer!

My time at Culture Amp was special. It was my first role after a decade of running Icelab with Max and Michael. Culture Amp hired everyone at Icelab after we decided to close the business, providing both a smooth transition and new opportunities to a singular group. I built a great working relationship with my manager, I was trusted to do big things, and I relished the chance to work with and learn from a large group of engineers. I’m deeply thankful for all of this.

Towards the end, I was serving as Culture Amp’s Director of Back End Engineering, and moving into engineering management. However, as any astute reader of this blog might attest, I am deeply motivated by hands on programming work, and all the learning and collaboration opportunities that go with it. I realised it was not the time to draw that chapter to a close (it might never!), and through that consideration I connected with Buildkite.

I’m excited to join Buildkite for many reasons! It’s a great Australian company with heart and personality. It brims with people I’ve long dreamt of working with. Developer tooling is an area close to my heart. And they’re growing a (majestic) Ruby app at the core of their tech. I can’t wait to dig in.

For me, this is also an intentional decision to stick with Ruby. The work I’m doing in Ruby OSS right now might be one of the biggest “dents in the universe” I get to make. I want to see this effort through, to complete our vision for Hanami 2.0, then learn from how it’s adopted by our community.

I have some time off between jobs, which I’ll use to give our Hanami work a real boost: I’ll be commiting nearly 6 weeks of full-time work to Hanami! Based on previous experience, this should see me get through what otherwise might have taken 6 months of part-time effort. I’m hoping this will get us significantly closer to 2.0. I’ll likely start another tweet thread of my efforts, so find me on Twitter if you’d like to follow along!

,

Ian BrownGCVE Backup and Disaster Recovery

Picking up where we left off last month, let’s dive into disaster recovery and how to use Site Recovery Manager and Google Backup & Protect to DR into and within the cloud with GCVE. But before we do, a quick advertisement: If you are in Brisbane, Australia, I suggest coming to the awesome Google Infrastructure Group (GIG) which focuses on GCVE where on 04 July 2022 I will be presenting on Terraform in GCVE.

,

Tim RileyOpen source status update, April 2022

April was a pretty decent month for my OSS work! Got some things wrapped up, kept a few things moving, and opened up a promising thing for investigation. What are these things, you say? Let’s take a look!

Finished centralisation of Hanami action and view integrations

I wrote about the need to centralise these integrations last month, and in April, I finally got the work done!

This was a relief to get out. As a task, while necessary, it felt like drudge work – I’d been on it since early March, after all! I was also conscious that this was also blocking Luca’s work on helpers all the while.

My prolonged work on this (in part, among other things like Easter holidays and other such Real Life matters) contributed to us missing April’s Hanami release. The good thing is that it’s done now, and I’m hopeful we can have this released via another Hanami alpha sometime very soon.

In terms of the change to Hanami apps, the biggest change from this is that your apps should use a new superclass for actions and views:

require "hanami/application/action"

module Main
  module Action
    # Used to inherit from Hanami::Action
    class Base < Hanami::Application::Action
    end
  end
end

Aside from the benefit to us as maintainers of having this integration code kept together, this distinct superclass should also help make it clearer where to look when learning about how actions and views work within full Hanami apps.

Enabled proper access to full locals in view templates

I wound up doing a little more work in actions and views this month. The first was a quickie to unblock some more of Luca’s helpers work: making access to the locals hash within templates work like we always expected it would.

This turned out to be a fun one. For a bit of background, the context for every template rendering in hanami-view (i.e. what self is for any given template) is an Hanami::View::Scope instance. This instance contains the template’s locals, makes the full locals hash available as #locals (and #_locals, for various reasons), and uses #method_missing to make also make each local directly available via its own name.

Luca found, however, that calling locals within the template didn’t work at all! After I took a look, it seemed that while locals didn’t work, self.locals or just plain _locals would work. Strange!

Turns out, this all came down to implementation details in Tilt, which we use as our low-level template renderer. The way Tilt works is that it will compile a template down into a single Ruby method that receives a locals param:

def compile_template_method(local_keys, scope_class=nil)
  source, offset = precompiled(local_keys)
  local_code = local_extraction(local_keys)

  # <...snip...>

  method_source << <<-RUBY
    TOPOBJECT.class_eval do
      def #{method_name}(locals)
        #{local_code}
  RUBY

Because of this, locals is actually a local variable in the context of that method execution, which will override any other methods also available on the scope object that Tilt turns into self for the rendering.

Here is how we were originally rendering with Tilt:

tilt(path).render(scope, &block)

My first instinct was simply to pass our locals hash as the (optional) second argument to Tilt’s #render:

tilt(path).render(scope, scope._locals)

But even that didn’t work! Because in generating that local_code above, Tilt will actually take the locals and explode it out into individual variable assignments:

def local_extraction(local_keys)
  local_keys.map do |k|
    if k.to_s =~ /\A[a-z_][a-zA-Z_0-9]*\z/
      "#{k} = locals[#{k.inspect}]"
    else
      raise "invalid locals key: #{k.inspect} (keys must be variable names)"
    end
  end.join("\n")
end

But we don’t need this at all, since hanami-view’s scope object is already making those locals available individually, and we want to ensure access to those locals continues to run through the scope object.

So the ultimate fix is to make locals of our locals. Yo dawg:

tilt(path).render(scope, {locals: scope._locals}, &block)

This gives us our desired access to the locals hash in templates (because that locals key is itself turned into a solitary local variable), while preserving the rest of our existing scope-based functionality.

It also shows me that I probably should’ve written an integration test back when I introduced access to a scope’s locals back in January 2019. 😬

Either way, I’m excited this came up and I could fix it, because it’s an encouraging sign of just how much of this view system we’ll be able to put to use in creating a streamlined and powerful view layer for our future Hanami users!

Merged a fix to stop unwanted view rendering of halted requests

Thanks to our extensive use of Hanami at Culture Amp, my friend and colleague Andrew discovered and fixed a bug with our automatic rendering of views within actions, which I was happy to merge in.

Shipped some long awaited dry-configurable features

After keeping poor ojab waiting way too long, I also merged a couple of nice enhancements he made to dry-configurable:

I then released these as dry-configurable 0.15.0.

Started work on unifying Hanami slices and application

Last but definitely not least, I started work on one of the last big efforts we need in place before 2.0: making Hanami slices act as much as possible like complete, miniature Hanami applications. I’m going to talk about this a lot more in future posts, but for now, I can point you to a few PRs:

  • Introducing Hanami::SliceName (a preliminary, minor refactoring to fix some slice and application name determination responsibilities that had somehow found their way into our configuration class).
  • A first, abandoned attempt at combining slices and applications, using a mixin for shared behaviour.
  • A much more promising attempt using a composed slice object within the application class, which is currently the base of my further work in this area.

Apart from opening up some really interesting possibilities around making slices fully a portable, mountable abstraction (imagine bringing in slices from gems!), even for our shorter-term needs, this work looks valuable, since I think it should provide a pathway for having application-wide settings kept on the application class, while still allowing per-slice customisation of those settings in whichever slices require them.

The overall slice structure is also something that’s barely changed since I put it in place way back in late 2019. Now it’s going to get the spit and polish it deserves. Hopefully I’ll be able to share more progress on this next month :) See you then!

,

Ian BrownGCVE Advanced Auto-Scaling

Let’s pick up where we left off from last months article and start setting up some of the features of GCVE, starting with Advanced Autoscaling. What is Advanced Auto-Scaling? Advanced Autoscaling automatically expands or shrinks a private cloud based on CPU, memory and storage utilisation metrics. GCVE monitors the cluster based on the metrics defined in the autoscale policy and decides to add or remove nodes automatically. Remember: GCVE is physical Dell Poweredge servers, not a container/VM running in Docker or on a hypervisor like VMware.

,

Tim RileyTwo years of open source status updates

Back in March of 2020, I decided to take up the habit of writing monthly status updates for my open source software development. 22 updates and 25k words later, I’m happy to be celebrating two years of status updates!

As each month ticks around, I can find it hard to break away from cutting code and write my updates, but every time I get to publishing, I’m really happy to have captured my progress and thinking.

After all, these posts now help remind me I managed to do all of the following over the last two years (and these were just the highlights!):

  • Renamed dry-view to hanami-view and kicked off view/application integration (Mar 2020)
  • Received my first GitHub sponsor (Apr 2020), thank you Benny Klotz (who still sponsors me today!)
  • Shared my Hanami 2 application template (May 2020)
  • Achieved seamless view/action/application integration (May 2020)
  • Brought class-level configuration to Hanami::Action (Jun 2020)
  • Introduced application-level configuration for actions and views (Jul 2020)
  • Added automatic inference for an action’s paired view, along with automatic rendering (Jul 2020)
  • Introduced application integration for view context classes (Jul 2020)
  • Supported multiple boot file dirs in dry-system, allowing user-replacement of standard bootable components in Hanami (Aug 2020)
  • Rebuilt the Hanami Flash class (Aug 2020)
  • Resumed restoring hanami-controller features through automatic enabling of CSRF protection (Sep 2020)
  • Added automatic configuration to views (inflector, template, part namespace) (Oct 2020)
  • Released a non-web Hanami application template (Oct 2020)
  • Started the long road to Hanami/Zeitwerk integration with an autoloading loader for dry-system (Nov 2020)
  • Introduced dedicated “component dirâ€� abstraction to dry-system, along with major cleanups and consistency wins (Dec 2020/Jan 2021)
  • Added support for dry-system component dirs with mixed namespaces (Feb/Mar/Apr 2021)
  • Released dry-system with all these changes, along with Hanami with working Zeitwerk integration (Mar/Apr 2021)
  • Ported Hanami’s app configuration to dry-configurable (May 2021),
  • Laid the way for dry-configurable 1.0 with some API changes (May/Jul 2021)
  • Returned to dry-system and added configurable constant namespaces (Jun/Jul/Aug/Sep/Oct 2021)
  • Introduced compact slice source dirs to Hanami, using dry-systems constant namespaces (Sep/Oct 2021)
  • Added fully configurable source dirs to Hanami (Nov/Dec 2021)
  • Shipped a huge amount of dry-system improvements over two weeks of dedicated OSS time in Jan 2022, including the overhaul of bootable components as part of their rename to providers, as well as partial container imports and exports, plus much more
  • Introduced concrete slice classes and other slice registration improvements to Hanami (Feb 2022)
  • Refactored and relocated action and view integration into the hanami gem itself, and introduced Hanami::SliceConfigurable to make it possible for similar components to integrate (Mar 2022)

This is a lot! To add some extra colour here, a big difference betwen now and pre-2020 is that I’ve been working on OSS exclusively in my personal time (nights and weekends), and I’ve also been slugging away at a single large goal (Hanami 2.0, if you hadn’t heard!), and the combination of this can make the whole thing feel a little thankless. These monthly updates are timely punctuation and a valuable reminder that I am moving forward.

They also capture a lot of in-the-moment thinking that’d otherwise be lost to the sands of time. What I’ve grown to realise with my OSS work is that it’s as much about the process as anything else. For community-driven projects like dry-rb and Hanami, the work will be done when it’s done, and there’s not particularly much we can do to hurry it. However, what we should never forget is to make that work-in-progress readily accessible to our community, to bring people along for the ride, and to share whatever lessons we discover along the way. The passing of each month is a wonderful opportunity for me to do this 😀

Finally, a huge thank you from me to anyone who reads these updates. Hearing from folks and knowing there are people out there following along is a huge encouragement to me.

So, let’s keep this going. I’m looking forward to another year of updates, and—checks calendar–writing April’s post in the next week or so!

,

Tim RileySalubrious Ruby: Don’t mutate what you don’t own

When we’re writing a method in Ruby and receiving objects as arguments, a helpful principle to follow is “don’t mutate what you don’t own.”

Why is this? Those arguments come from places that we as the method authors can’t know, and a well-behaved method shouldn’t alter the external environment unexpectedly.

Consider the following method, which takes an array of numbers and appends a new, incremented number:

def append_number(arr)
  last_number = arr.last || 0
  arr << last_num + 1
end

If we pass in an array, we’ll get a new number appended in the returned array:

my_arr = [1, 2]
my_new_arr = append_number(my_arr) # => [1, 2, 3]

But we’ll also quickly discover that this has been achieved by mutating our original array:

my_arr = [1, 2]
my_new_arr = append_number(arr) # => [1, 2, 3]
my_arr # => [1, 2, 3]

We can confirm by an object equality check that this is still the one same array:

my_new_arr.eql?(my_arr) # => true

This behavior is courtesy of Ruby’s Array#<< method (aka #append or #push), which appends the given object to the receiver (that is, self), before then returning that same self. This kind of self-mutating behaviour is common across both the Array and Hash classes, and while it can provide some conveniences in local use (such as a chain of #<< calls to append multiple items to the same array), it can lead to surprising results when that array or hash comes from anywhere non-local.

Imagine our example above is part of a much larger application. In this case, my_arr will most likely come from somewhere far removed and well outside the purview of append_number or whatever class contains it. As the authors of append_number, we have no idea how that original array might otherwise be used! For this reason, the courteous approach to take is not to mutate the array, but instead create and return a new copy:

def append_number(arr)
  last_number = arr.last || 0

  # There are many ways we can achieve the copy; here's just one
  arr + [last_number]
end

This way, the caller of our method can trust their original values to go unchanged, which is what they would likely expect, especially if our method doesn’t give any hint that it will mutate.

my_arr = [1, 2]
my_new_arr = append_number(arr) # => [1, 2, 3]
my_arr # => [1, 2]

This is a very simple example, but the same princple applies for all kinds of mutable objects passed to your methods.

A more telling story here comes from earlier days of Ruby, around how we handled options hashes passed to methods. We used to do things like this:

def my_method(options = {})
  some_opt = options.delete(:some_opt)
  # Do something with some_opt...
end

my_method(some_opt: "some value")

Using trailing options hashes like this was how we provided “keyword arguments” before Ruby had them as a language feature. Now the trouble with the method above is that we’re mutating that options hash by deleting the :some_opt key. So if the user of our method had code like this:

common_options = {some_opt: "some value"}

first_result = my_method(common_options)
second_result = my_method(common_options)

We’d find ourselves in trouble by the time we call my_method the second time, because at that point the common_options hash will no longer have some_opt:, since the first invocation of my_method deleted it — oops!

This is a great illustration of why modern Ruby’s keyword arguments work the way they do. When we accept a splatted keyword hash argument like **options, Ruby ensures it comes into the method as a new hash, which means that operations like options.delete(:some_opt) do in fact become local in scope, and therefore safe to use.

So now that we’ve covered both arrays and hashes now as Ruby’s most common “container” objects, what about the other kinds of application-specific structures that we might encounter in real world codebases? Objects representing domain models of various kinds, an instance of an ActiveRecord::Base subclass, even? Even in those cases, this principle still holds true. Our code is easier to understand and test when we can reduce the number of dimenstions to its behaviour, and mutating passed-in objects is a big factor in this, especially if you think about methods calling other methods and so on. There are ways we can design our applications to make this a natural approach to take, even for rich domain objects, but that is a topic for another day!

Until then, hopefully this walkthrough here can serve as a reminder to keep our methods courteous, to respect mutable values provided from outside, and wherever possible, leave them undisturbed and unmutated. Salubrious!

Bonus content! In preparing this post, I thought about whether it might be helpful to note that Ruby is a “pass by reference” language, since that’s the key underlying behavior that can result in these accidental mutations. However, my intuition here was actually back to front! Thanks to this wonderful stackoverflow answer, I was reminded that Ruby is in fact a “pass by value” language, but that all values are references.

,

Tim SerongGo With The Flow

We installed 5.94kW of solar PV in late 2017, with an ABB PVI-6000TL-OUTD inverter, and also a nice energy efficient Sanden heat pump hot water service to replace our crusty old conventional electric hot water system. In the four years since then we’ve generated something like 24MWh of electricity, but were actually only able to directly use maybe 45% of that – the rest was exported to the grid.

The plan had always been to get batteries once we are able to afford to do so, and this actually happened in August 2021, when we finally got a single 10kWh Redflow ZCell zinc bromine flow battery installed. We went with Redflow for several reasons:

  • Unlike every other type of battery, they’re not a horrible fire hazard (in fact, the electrolyte, while corrosive, is actually fire retardant – a good thing when you live in a bushfire prone area).
  • They’re modular, so you can just keep adding more of them.
  • 100% depth of discharge (i.e. they’re happy to keep being cycled, and can also be left discharged/idle for extended periods).
  • All the battery components are able to be recycled at end of life.
  • They’re Australian designed and developed, with manufacturing in Thailand.

Our primary reasons for wanting battery storage were to ensure we’re using renewable energy rather than fossil fuels, to try to actually use all our local power generation locally, and to attain some degree of disaster resilience.

Being in Tasmania, most of our grid power is from renewable sources anyway (hydro), so the renewable energy argument may seem a little weak at first, unless you cast your mind back to the time some idiot decided to deplete our dams by selling a whole lot of hydro-generated power to Victoria in an El Niño year, then the Basslink cable broke and Tasmania had to fire up a bunch of diesel generators to get through winter. Good times.

On the local generation and use front, as I mentioned at the start of this post, we’ve previously exported more than half the energy we generated, but the feed-in tariff we get from Aurora (the power company) is only $0.06501 per kWh. Compare that to the rate we pay for grid energy ($0.29852/kWh peak or $0.139/kWh off-peak). Say we exported 13.2MWh in the last four years, we would have received about $858 worth of credit… But then when we drew power back from the grid at night, or on cloudy days, we would have paid somewhere between $1834-$3940 for that same amount of power. Treating the grid as a proxy for battery storage does not make any sort of financial sense.

As for disaster resilience, we don’t often have grid outages, but we do have them, and that can be a problem. Notably, all our potable water comes from rainwater tanks attached to the house and shed, and the pumps that push that water to the taps are electric, so if the grid is down we don’t have running water. Sure, I can get water out of the tanks with a bucket or a jug, and that’s fine for a little drinking or handwashing, but it’s not good long term. Then there’s our fridges and freezer – at any given time we’re likely to have a lot of stored frozen meat from animals we farm. We don’t want to lose that in a potential extended grid outage, as could happen in bushfire season or during a severe weather event. Also, it’s nice to still have power for our local network and NBN kit, so we can check the TasNetworks Power Outages page to find out WTF is going on.

Complete grid independence would be nice, and with our current power utilisation and a single Redflow battery we could almost do it in the height of summer, or even on some good days in spring or autumn, where we can generate all we need to run the house during the day and charge the battery to 100%, then draw it down overnight. The kink is that Redflow batteries need to undergo a maintenance cycle at least every three days where they are completely discharged for a few hours. If you’re grid connected you don’t really notice this, because the maintenance cycle commences at sunset and once the battery is drained you’re just using grid power again until the sun comes up, but it does mean we can’t be grid independent even if we theoretically have enough PV generation to do so, until we get a second battery (with more than one ZCell, the maintenance cycles are interleaved so at least one battery will always have some power in it).

The other problem with grid independence is that as much as Tasmania is excellent for solar PV generation in summer, it sucks in winter. Looking at our generation and usage figures for 2019, from mid-May to mid-August we were only able to generate 17% of the power we used, and I’ve seen days where we only generated 1-2kW in the entire day. Compare that with summer when we’ve peaked above 40kW some days in December.

Still, if the grid went away for a long time in the warmer half of the year with our current setup, it’d be irritating every few nights, but I reckon we’d manage OK. Of course, there would be some adjustments required to minimise our utilisation: I’d set the blockout timer on the Sanden so the hot water only heated during daylight hours, I’d turn most of our computer equipment off overnight, we’d try to avoid using the microwave at the same time as any other chunky electrical appliance so as not to pull more than 3kW continuously from the ZCell, there’s some lights we usually leave on that we’d just turn off, and so forth. In the colder half of the year, well, I guess we’d try to eat all the frozen food quickly then limp along as best as possible. We would still have some power, some of the time.

When we originally had the PV installed, it was AC coupled, i.e. the solar panels were connected to the inverter, and the inverter was connected to the grid, along with our loads. Our choice of inverter (ABB) was partly because it was Selectronic certified, and at the time we knew Redflow to be working with Selectronic battery inverters. Once we finally got to contacting Murray Roberts from Lifestyle Electrical Services in order to get a quote and talk about installation, circumstances had changed. It turns out that the Selectronic kit just doesn’t like flow batteries with their need to be completely discharged periodically. Victron Energy gear on the other hand works really well with – and is fully supported for use with – Redflow’s ZCell batteries. Lesson learned: with changing technology it doesn’t always pay to plan too far in advance.

Murray initially proposed a setup which would have hooked straightforwardly into our AC coupled solar, i.e. the ABB inverter and PV cells would remain connected as is, then there’d be a Victron energy meter to measure what was coming from the PV and from the grid, a Victron MultiPlus II inverter/charger to charge the battery and pull from the battery to run some loads, plus a Victron Cerbo GX and GX Touch to provide monitoring and control. Some of our power circuits to the house would be hooked up as Essential House Loads (i.e. to be supported by the battery during a grid outage), and some would be Non-Essential House Loads, i.e. powered by the grid and/or PV, but without battery backup. The Victron Cerbo GX and the Redflow ZCell Battery Management System (BMS) are internet connected to hook up with Victron’s VRM portal, which provides handy monitoring tools and graphs, and to allow remote support and assistance and firmware updates. The initial proposal looked something like this:

AC Coupled Solar schematic. Some detail missing (e.g. isolators) but you get the idea.

That configuration would have involved the least messing around, and met our goals of:

  • Utilising as much of your own energy as possible locally.
  • Dealing with occasional/unexpected grid outages (modulo the ZCell maintenance cycle).

But, it still fundamentally relied on the grid. With AC coupled solar, if the grid goes down your inverter automatically goes into anti-islanding mode, and won’t give you any power from your PV array even if the grid is down during the day and the sun is shining on your panels. Your first thought here may be “wait, if the grid is down but I still have sunlight, surely I should still have power”, and that’s an understandable reaction, but anti-islanding is actually a safety feature. If the juice goes from the PV cells to the inverter to the grid and your loads, and you’re exporting power while a power company employee is doing maintenance works on the grid side, you could electrocute them. This would not be a good thing.

A perhaps less obvious problem with this setup is that you can’t black start the site during an extended grid outage. If the grid is down and the inverter is thus in anti-islanding mode, you have no way to get power to restart the system and recharge the battery from empty (unless you’ve also got a generator). “But the grid never goes down for that long…” you might say, until you look at the outages really severe bushfires can cause (think: East Gippsland in the 2019-2020 Black Summer bushfires).

After some further discussion, Murray proposed getting rid of the ABB inverter, and doing DC coupled solar instead, with a Victron MPPT RS hooked up to the PV, two MultiPlus IIs, so that we could handle up to 10kW of power (that’s the maximum limit of all loads and grid export), with all house loads hooked up as Essential, so they can be supplied by whatever combination of grid, solar and battery power is available at any given time. Voilà:

DC Coupled Solar schematic. Again, some detail missing, but mostly accurate.

One thing that’s missing in the above diagram is a manual changeover switch we had installed later, so that if there’s ever a fault in the Victron cluster that requires major works, but the grid is still up, we can manually switch all our loads back over to the grid side for the duration. Not that I expect to need that functionality, but better to have it than not just in case.

The 10kW maximum has proven to be fine, by the way – we just don’t ever put anything like that much power through the system at once. During the day we might pull 400-700W continuously with spikes from 1-3kW or occasionally 4-5kW when multiple heavier loads come on. I think the highest we’ve managed was 7kW very briefly one time with a panel heater and the microwave and the hot water and the clothes dryer and gods only know what else was on at the time, the point is it’s not easy to get the load that high. Note that we currently have a gas stove and oven – if we switched that to electric we might want to be a bit cautious about running lots of other heavy loads while cooking, but I suspect we’d still be fine in general.

Rhys, one of Murray’s crew, did a fantastic job of installing all the kit over several days, then Murray came out to do the final commissioning and bring the system online on August 31, 2021. Here’s a couple of pictures:

RedFlow ZCell battery, 2x Victron MultiPlus IIs, Victron MPPT RS
MPPT again, with ethernet switch, sub board, Victron Cerbo GX and ZCell BMS off to the right.

The transparent box in the above photo contains the Cerbo GX and the ZCell BMS, along with a little 12V backup power supply so that those things keep running if the grid fails and the ZCell is at 0% State of Charge (SoC). The ethernet switch above the sub board hooks up to the network point I installed previously when I was using a Raspberry Pi to monitor the ABB inverter’s generation. It’s Power-over-Ethernet, run from a UPS in my office which also keeps the NBN box and router alive, so the whole system still has internet access for about an hour even if all other power sources are dead (handy if there were some sort of fault and we needed remote assistance).

Here’s what the main switchboard looks like now. The leftmost switch (“Main switch (grid supply)”) turns grid power on/off to the sub board under the house, which goes from there to the Multis’ AC input. The AC out comes back up here to the right-hand switch (“Victron Sub Board Backup Supply”) and thence to all the loads (the various switches in the middle).

Do not fuck with the electricity.

Inside the house there’s a neat little touchscreen console (the Victron GX Touch), which connects via an HDMI cable in the wall to the Cerbo under the house. This shows you what everything is doing at any given time, provides notifications of alarms (e.g.: “Grid Lost”) and has a series of menus for configuring the system. The exact same console is also accessible via a web browser or mobile phone, either over the local network, or remotely via the VRM portal.

Three days after everything was up and running, we went out to run some errands and came back home in the afternoon to discover the Cable PI device in our kitchen beeping like mad. All the power was still on in the house. I called Murray in a bit of a panic thinking something was broken, but it turned out we were experiencing an actual grid outage over a large area – all the way from Grove to Leslie Vale, 802 properties were without power due to wires down in strong wind, and the PV was powering our house. We didn’t feel a thing. The UPS in my office noticed a small dip for a second or two when the grid failed, but the microwave clock was still on, and other computer gear not hooked up to a UPS remained up during the cutover. And the battery kept on charging – it was at 65% SoC when the grid went out and up to 83% by the time the grid came back.

This was just awesome.

A few days after that, suddenly, at 19:39 on September 6, we were without power. The grid was up, but the Victron kit had shut down. This was not awesome. It happened during the ZCell’s regular maintenance cycle, and at the time we got a warning in the BMS logs, and a battery high voltage warning from the MPPT. It was unclear then exactly what the problem was though – our MPPT RS was a newer model so maybe it was different somehow from what had been previously tested? Also, we discovered the DVCC setting still needed to be turned on, so maybe that was the issue. Anyway, I power cycled the Victron kit and everything was fine again until a couple of weeks later on September 19, when the system shut down again during a maintenance cycle, and again coinciding with battery high voltage warnings.

Because of the previous shutdown, Murray had been in contact with Simon Hackett from Redflow, and Simon subsequently enabled the “DC-coupled PV – feed in excess” setting. The assumption here was that the extra power being delivered from the ZCell during the maintenance discharge wasn’t absorbed by our house loads, i.e. it was trying to discharge at 1kW, but our loads were utilising less than that, and there was nowhere for the excess to go, hence the shutdown. Enabling “DC-coupled PV – feed in excess” allows power from the DC bus to be sent to the grid if necessary, which turns out to be the case at our site. A second ZCell would mitigate this because the one under maintenance would simply be able to dump to the other (assuming it wasn’t already full), but we only have one battery so far.

At this point, after those two final settings changes (and a firmware update) the system was operating exactly as it should. Everything was configured correctly, and we could survive grid outages if there was charge in the battery and/or the sun was up. Our electricity meter was replaced on September 21 so we could switch to Tariff 93 Peak & Off-Peak billing. There were no further unexpected shutdowns. Everything was totally fine, except we were still seeing these weird battery high voltage warnings during the ZCell maintenance cycle, reported by both the BMS and the MPPT.

MPPT Alarm #2 Battery high voltage

This is something neither Murray nor Simon had seen before. What followed was several weeks of troubleshooting and analysis, which I found absolutely fascinating.

I was keeping detailed notes of what happened during each maintenance cycle, and what I saw in the BMS logs, and on the Cerbo console. Several maintenance cycles later I discovered a correlation between the battery high voltage warnings, and sudden large changes in AC loads, notably when a 2400W panel heater in our bedroom turned itself on and off overnight. Also, the high voltage warnings seemed to be more likely to occur if we went into maintenance with a high state of charge, versus a low state of charge.

Overnight load spikes

Simon, who knows and understands how the ZCell behaves during maintenance, explained about the Energy Extraction Device, which is part of the unit whose purpose is to deliberately drain the battery down to zero for maintenance in a timely fashion. He wondered if there was some issue where very high power demands at short notice, while the the EED was active, were causing the DC bus voltage to fluctuate and in turn cause the MPPT to respond in an unusual manner. We experimented with changing settings on the BMS to activate the EED later than usual in the maintenance cycle (“Start maintenance when SoC below x%”), and also experimented with limiting inverter output on the Multis, and tweaking the maximum charge voltage.

After a few more cycles of observation, the suspicion became that the MPPT itself wasn’t at fault, rather it was just being the messenger. Maybe these odd voltage spikes had always happened at other sites too, but the new MPPT RS units were doing a better job of noticing them?

Later in October, Simon noticed that we were seeing spikes very close to when the battery was nearly completely empty. At that point in time, the EED was telling the Multis that it had a 10 amp output capacity, but if the Multis tried to draw on that to handle a sudden rise in load (as from our panel heater for example), the battery voltage would collapse, and the system would oscillate a bit between those two states. That behaviour was fixed with a small firmware change which I believe later landed in the BMS 1.1.11 release. Unfortunately, that change by itself didn’t make the high voltage warnings go away.

A few days after that, we suddenly had a slew of #201 – Internal DC voltage errors from the MPPT, so we were back to being concerned that maybe there was something wrong with that piece of equipment, especially given those errors began to crop up more often as time passed.

MPPT Alarm #201 – Internal DC voltage error.

Victron’s documentation ominously stated that this error meant “a measurement circuit inside the unit is broken” and the unit “is really broken, not safe for use, and if it hadn’t stopped working already then it would have stopped working soon”. Clearly a replacement or repair was in order, but I’ll get to that later.

On November 4, looking at the logs I’d collected from November 2, Simon noticed that one of the high battery voltage warnings happened at quite a high state of charge (72%), which meant it wasn’t really about the battery running out of energy and having the voltage collapse. It looked like it was the EED being over-drawn, regardless of how much energy was still in the battery. It turns out there’s a thing that the ZCell does to handle surge demand when the EED is on, called an “EED switchback”. ZCells internally have three contactors, for Charge, Discharge and EED (also known as Strip). In normal operation, the C and D contactors are on, and E is off, so the battery can be charged or discharged at will, and the EED is doing nothing. During maintenance, the EED comes on, but it can’t deliver more than 20 amps. If the site pulls more than the EED can supply, the battery goes back to normal operation (C and D on, E off) while the high demand is present. Once the high demand goes away, it switches the EED back on, to keep discharging at the normal rate of 1kW. But, by default, that switchback process only happens five times per battery maintenance cycle so as to avoid the potential for excessive cycling of the contactors in weird edge cases.

Looking at our overnight load with the panel heater on, there’s loads of spikes from below 1kW to above 1kW, so we were getting through that switchback limit very quickly. After that, with the high load from the panel heater, it was entirely possible that the Multi cluster would try to pull more power than the EED could provide, and the EED would shut down in response, resulting in weirdness on the DC bus. Simon’s suspicion for why this hadn’t been seen before was that it needed at least four casual factors to be present at once:

  1. Site demand is spiky for the entire night.
  2. The spikes start below 1kW and end above 1kW to use up the switchback quota.
  3. The site has only one battery (if a second battery were present it would handle the surge load while the first was in maintenance).
  4. The two Multis are capable of running far more load than the battery can service.

If we were to change or remove any one of those factors, we wouldn’t see the problem. So, as a test, Simon changed the switchback limit from 5 to 50, and I watched what happened during the next maintenance cycle. The status page of the BMS web interface shows, among other things, the current state of the contactors. Here’s an example with C and D on, and E off:

The ZBM logs also show the contactor state over time. Here’s a snippet I’ve colourised to make the state changes obvious:

If we take the above section of ZBM logs from 2021-11-08 23:15 to 23:29, it looks like we had “_ D E” up until 23:17, then switched to “C D _” from 23:18 to 23:26, then finally to “_ _ E” at 23:27. Based on this I’d imagine we had one switchback event that lasted eight minutes. But I had earlier noticed on the status page that the contactors seemed to be toggling more rapidly, so I wrote a little script to scrape the BMS REST API once per second and dump that to a file, which shows Charge and EED toggling on/off about ten times in that window:

2021-11-08T23:14:28+11:00    "_ D E"
2021-11-08T23:17:59+11:00    "C D _"
2021-11-08T23:18:29+11:00    "C D E"
2021-11-08T23:18:34+11:00    "_ D E"
2021-11-08T23:18:56+11:00    "C D _"
2021-11-08T23:19:14+11:00    "C D E"
2021-11-08T23:19:20+11:00    "_ D E"
2021-11-08T23:19:54+11:00    "C D _"
2021-11-08T23:20:17+11:00    "C D E"
2021-11-08T23:20:18+11:00    "_ D E"
2021-11-08T23:20:49+11:00    "C D _"
2021-11-08T23:21:19+11:00    "C D E"
2021-11-08T23:21:22+11:00    "_ D E"
2021-11-08T23:21:46+11:00    "C D _"
2021-11-08T23:21:51+11:00    "_ D _"
2021-11-08T23:21:54+11:00    "C D _"
2021-11-08T23:22:23+11:00    "C D E"
2021-11-08T23:22:26+11:00    "_ D E"
2021-11-08T23:22:43+11:00    "C D E"
2021-11-08T23:22:45+11:00    "C D _"
2021-11-08T23:23:26+11:00    "C D E"
2021-11-08T23:23:30+11:00    "_ D E"
2021-11-08T23:23:43+11:00    "C D _"
2021-11-08T23:24:30+11:00    "C D E"
2021-11-08T23:24:34+11:00    "_ D E"
2021-11-08T23:24:43+11:00    "C D _"
2021-11-08T23:25:35+11:00    "C D E"
2021-11-08T23:25:39+11:00    "_ D E"
2021-11-08T23:25:49+11:00    "C D _"
2021-11-08T23:26:40+11:00    "C D E"
2021-11-08T23:26:44+11:00    "_ D E"
2021-11-08T23:26:49+11:00    "_ _ E"

On the assumption we were hitting way more switchbacks than expected, Simon just went and set the maximum switchback count to 999. A few days later, taking the log from my script, I saw something like 150-160 switchbacks, but given we’d set the limit way high, that fixed all the high voltage warnings, except for one, right at the very end of the maintenance cycle when the discharge limit from the ZCell drops from 10 amps to 0 amps.

Simon discussed this final spike with the folks who built the EED, and found that when current draw from the EED stops, there is a voltage spike, of very low energy, for 10ms, and it can rise as high as 64V during that period before dropping back to the expected 57V. It’s normal for the EED to do this, and as there’s no real energy in it, it won’t be damaging anything. The thing about our site seems to be the new MPPT RS, with a new voltage sensing circuit that’s actually capable of noticing the spike, whereas the other gear (the MultiPlus IIs) misses it because it’s so short. The advice from the electrical engineer was to try adding more capacitors on the DC bus to absorb the spike. We already had two 47,000uF capacitors on there, so Murray went and ordered two more.

With the high battery voltage warnings out of the way, we were back to the #201 Internal DC voltage errors from the MPPT. On the assumption the unit was indeed faulty in that regard we requested a replacement, but Victron came back and said the problem could be fixed by replacing two resistors on the main board of the unit. I guess that makes sense – if you can fix a problem with $2 worth of resistors, that’s three orders of magnitude cheaper than replacing the whole unit.

By then we were getting into December, and what with pandemic-related shipping delays and the Christmas holiday period, it was later in January before we were able to get the additional capacitors installed on the DC bus, and replace the resistors in the MPPT’s voltage sensing circuit. The additional capacitors went just fine, the replacement resistors not so much. Once the MPPT was powered back up it claimed there was zero volts coming from the PV, even though the sun was shining, and the LCD display started flickering strangely. Something was definitely broken here, so we powered it back down, and Simon arranged for a replacement unit to be sent out, which took another few weeks, which is a damn shame in January/February, being prime solar PV generation time.

The delay did however allow me to spend some time messing around with scheduled charges to see if there was a cost benefit to grid-charging the battery during off-peak times, then drawing it back down during peak, because the reality is we’re going to want to do this in winter when there’s not much sun, so why not try it out in advance? TL;DR: Yes, it’s worth grid charging the battery off-peak, provided you use all that power during peak times, but it’s a bit irritating trying to figure out exactly what you’ll save. In one of my tests it was the difference between paying $3.85 for about 20kWh of usable electricity in a 24 hour period versus paying $4.70, so it’s not insignificant.

Rhys came out and installed the replacement MPPT on February 11, and was done by the middle of the day. Everything was running beautifully again, but when the unit came online there were ten instances of the dreaded #201 Internal DC Voltage Error, along with a #27 Charger Short Circuit. I used the VictronConnect app on my phone to see if I could get any more information directly from the MPPT. It told me there was a firmware update available from v1.05 to v1.08, so I went looking for information about that, and discovered that Victron’s error code documentation had been updated since I first saw it back in late October. In addition to the ominous warnings about broken measurement circuits it now also said:

“Make sure to update the firmware to at least v1.08, in previous firmwares the limits were too strict. And it could trigger falsely during MPPT start-up in the morning and MPPT shutdown in the evening.”

So I updated the firmware, and writing this now, two and a half months later, we’ve not seen a single #201 since. Could this have always been a firmware issue? Maybe, given the “accepted answer” on this Victron Community forum post says that firmware version v1.08 “solves the vast majority of MPPT RS, and Inverter RS, Error 201 issues”. Or maybe it was both – maybe we had a broken bit of kit and broken firmware too. Either way, it’s fixed now.

I continued to monitor regular maintenance cycles, and also deliberately forced maintenance a couple of times with a high state of charge to try to stress it as much as I could. During those periods I saw something like 3-10 switchbacks, so Simon set our switchback limit back down from 999 to 30. I understand a future Redflow firmware update will change this default for everyone to somewhere between 25-50, and I’m very happy that this unexpected testing at our site resulted in firmware improvements that will presumably benefit other ZCell users too.

By late April we’d been through 33 maintenance cycles since the extra capacitors went in, with 26 of those occurring since the new MPPT was installed. There had been only three occasions when the BMS briefly noticed alleged high battery voltages in that time. The MPPT was completely silent until April 20 when we got three battery high voltage warnings within an 8 minute interval right at the end of the maintenance cycle, when the battery was almost completely empty. But the weather had also started to get cold, and those errors coincide with a spike from our panel heater, which is consistent with our earlier observations about load spikes with the EED on being “difficult” and really just points to replacing the panel heater with a heat pump. Heat pump are way more energy efficient, have much smoother load, and can also be used for cooling in summer (they’re called “reverse cycle air conditioners” on the mainland).

That’s about the end of the story. The system is brilliant, and we could not be happier with the support we’ve received from Simon at Redflow, who’s been extremely generous with his time and knowledge, and Murray and Rhys of Lifestyle Electrical Services. Thanks for everything guys, I’ve learned a lot. In the eight months the system has been running we’ve generated 4631kWh of electricity and “only” sent 588kWh to the grid, which means we’ve used 87% of what we generated locally – much better than the pre-battery figure of 45%. I suspect we’ve reduced the amount of power we pull from the grid by about 30% too, but I’ll have to wait until we have a full year’s worth of data to be sure. We’ve also survived or shortened at least five grid outages with durations from a few minutes to a few hours.

The next thing to do is get a second ZCell, and possibly eventually think about a third. Given our current generation capability, two ZCells would allow us to store and utilise 100% of our generated power locally. We’d also have the ability to handle grid outages at any time, because with two batteries the maintenance cycles interleave and they can be configured to always ensure there’s a minimum amount of charge somewhere. A third would allow us to look at Standby Power System (SPS) mode where one battery is fully charged, then put into hibernation where it can remain for months. This sounds like a great way to have backup storage available for grid outages in the middle of winter when there’s no sunlight.

Appendix A – Settings Worth Messing With

Scheduled charging on the Cerbo GX console

In summer I scheduled charges of the battery to 15% between the hours of 04:00 and 08:00, and 30% between the hours of 15:00-17:00. Peak electricity hours during daylight savings are 08:00-11:00 and 17:00-22:00, and I found that a 15% charge overnight from the grid was enough to have a bit in the battery for the morning peak before the PV really got going. The afternoon charge was there mostly just in case we had a cloudy day – we’d usually get way more charge than that from the sun anyway. Now that we’re off daylight savings, peak hours change to 07:00-10:00 and 16:00-21:00, so I’ve set it to a 30% charge from 03:00-07:00 and a 50% charge from 14:00-16:00, which seems to be about right given our general peak utilisation and decreasing sunlight. I’m unlikely to set the afternoon charge higher than 50% because I don’t want to potentially go into a maintenance cycle with the battery very full, but I may re-evaluate that as we get deeper into winter.

It’s worth mentioning that during a scheduled charge, the power will come from wherever the Victron gear can find it, so if the sun is shining, you’ll be charging from the PV, not the grid. One thing to note is that during the scheduled charge period, the battery will not be used to support loads, even if it’s currently got a higher SoC than your limit. Some power will trickle away slowly though, I assume to run the pumps and supporting electronics of the battery itself.

My choice of timing for the overnight charge (four hours up to the start of the morning peak) is me wanting to have some power in the battery for as long as possible overnight in case of outages, without potentially interfering with maintenance cycles, which typically will have finished some time in the wee hours.

I also set up a 15% charge on weekend mornings. This doesn’t save us any money at all (actually it’ll be costing us a couple of tens of cents) because weekends are all off-peak power. The reason is again to have some opportunistic grid backup. Before I set this up we had an outage at 07:45 one Saturday morning with the battery empty, and had to wait until about 09:30 for the PV to bring the battery up to 10% again before everything came back online. Still, that then got us through the remainder of the grid outage which finished at about 10:15.

Battery Maintenance Settings on the ZCELL BMS

On the Battery Maintenance screen of the ZCell BMS, I’ve got “Immediate maintenance for batteries with an EED” turned off, and “Start Maintenance When SoC Below 25%” enabled. This is to try to reduce the amount of time the EED runs, to limit switchbacks caused by our spiky load. In summer I also set “Daily SoC Limit Before Maintenance” to 50%, so the battery would not let itself be charged more than half way on those long hot days with late sunsets and early sunrises. This was to minimise maintenance cycle time, because I’d previously seen occasions where we went into maintenance with 100% SoC, and the cycle didn’t finish before the following morning when the sun came up. I also had a couple of times where I guess some timeout expired and the ZCell went into its final chemical maintenance state while it still had a few percent of charge. Not letting it get very full on maintenance days avoids these situations. Now that we’re getting towards winter I’ve removed that limit because the nights are longer and I expect our evening power utilisation to be higher, i.e. we should naturally use up whatever power we’re able to generate in plenty of time during winter maintenance cycles.

It’s also worth checking the latitude and longitude are set correctly on the Site Configuration page, because that’s how the BMS figures out when the sun sets and thus when to start maintenance by default.

Appendix B – VRM Portal

The VRM portal is a remote monitoring and management web interface which Victron provides gratis for users of their hardware. It provides a realtime view of the same live utilisation information you can get from the Cerbo console, plus handy graphs of solar, grid and battery consumption.

Consumption 2022-04-26. Red is grid, yellow is solar, blue is battery.

It also provides detailed graphs of just about anything you can think of from any of the system components. It’s extremely useful. Without this I never would have been able to correlate the battery high voltage warnings with load spikes and changes in the ZCell discharge current limits.

Viewing a bunch of interesting detail all at once

The data for the advanced graphs is stored for at least six months, and the solar yield and consumption data is stored for at least 5 years. The alarm logs don’t hang around that long – I suspect it may just be showing the last 1000 entries. Somewhat irritatingly, most of these are usually low battery alarms that we don’t care about (you see a lot of them during maintenance cycles).

Appendix C – Security / Connectivity / Internet Access

The ZCell BMS and Victron Cerbo GX both need to be connected to the internet for firmware updates, remote support, and to work with the VRM portal. They don’t need to be connected 100% of the time, but they do want the connection for those reasons. The system will operate just fine if the internet is down though, and you don’t have to use the VRM portal if you don’t want to. I’ve put everything on a separate network, so I can access the BMS and the Cerbo console from my desktop/laptop/phone, but the BMS and Cerbo can’t do the reverse. It’s not that I don’t trust Redflow or Victron, it’s just sensible to keep systems that allow any form of remote access isolated from the rest of your internal network.

The BMS and Cerbo both provide WiFi APs for initial configuration. I’ve since turned those off. I can use the wired connection to the BMS to turn the WiFi back on if I ever need it, and I can do the same for the Cerbo from its console.

The Cerbo and MPPT both speak Bluetooth, so you can use the VictronConnect app to talk to them from your phone, to view status and update firmware.

Appendix D – Hackability

The ZCell BMS has a REST API, which is documented in the online help available from its web interface. This is how I was able to write a few scripts to log the battery state of charge, the contactor state, and the voltage and warning indicator status:

A bunch of Victron stuff is open source, notably Venus OS, which is the software that runs on the Cerbo. It looks fairly straightforward to get root on these things. It’s also possible to hook up the Victron kit to Home Assistant. I haven’t tried actually doing any of these things yet myself.

Appendix E – Aurora Plus

Having switched to Tariff 93 and gotten a fancy new electricity meter, we were able to use the Aurora Plus service from the power company. This provides a web interface and mobile phone app for viewing your power usage down to the hour, colour coded to indicate peak and off-peak usage and solar feed in. You also get a monthly, rather than quarterly bill. This all sounded pretty neat, so I signed up.

Aside from having used it to confirm that the figures I get from the VRM Portal and the power company actually match, it’s turned out to not be especially great.

While the electricity meter records usage information every 15 minutes, it’s only sent back to Aurora once per day, so the usage data is never actually live. Sure, you can see history, but this is useless for adjusting your power consumption on the fly. Compare that with the VRM Portal or Cerbo console, where I can see at a glance how much power is being used and how much solar is being generated right now and decide to turn appliances on or off appropriately.

Also, it nags you to give them money. It’s continually telling me I have a big red negative dollar balance, and periodically notifies me to “top up now to get ahead of your monthly bill”. No. I will pay the bill by the due date listed on the bill, after the bill actually arrives.

Finally, it costs eleven cents a day for the privilege of having the service. Under the circumstances I think I’m going to cancel it and just go back to quarterly billing.

Aurora+, trying to trick me into paying in advance.

Update 2022-06-17: I never got around to cancelling (you can’t do it online – you actually have to call and speak to someone), but I just received a notification saying “From 1 July 2022 you will no longer be charged 11c/day for aurora+”, so I’ve decided to keep it.

,

BlueHackersFree psychologist service at conferences: April 2022 update

We’ve done this a number of times over the last decade, from OSDC to LCA. The idea is to provide a free psychologist or counsellor at an in-person conference. Attendees can do an anonymous booking by taking a stickynote (with the timeslot) from a signup sheet, and thus get a free appointment.

Many people find it difficult taking the first (very important) step towards getting professional help, and we’ve received good feedback that this approach indeed assists.

So far we’ve always focused on open source conferences. Now we’re moving into information security! First BrisSEC 2022 (Friday 29 April at the Hilton in Brisbane, QLD) and then AusCERT 2022 (10-13 May at the Star Hotel, Gold Coast QLD). The awesome and geek friendly Dr Carla Rogers will be at both events.

How does this get funded? Well, we’ve crowdfunded some, nudged sponsors, most mostly it gets picked up by the conference organisers (aka indirectly by the sponsors, mostly).

If you’re a conference organiser, or would like a particular upcoming conference to offer this service, do drop us a line and we’re happy to chase it up for you and help the organisers to make it happen. We know how to run that now.

In-person is best. But for virtual conferences, sure contact us as well.

The post Free psychologist service at conferences: April 2022 update first appeared on BlueHackers.org.

,

FLOSS Down Under - online free software meetingsApril Hack Day Report

The hack day didn’t go as well as I hoped, but didn’t go too badly. There was smaller attendance than hoped and the discussion was mostly about things other than FLOSS. But everyone who attended had fun and learned interesting things so generally I think it counts as a success. There was discussion on topics including military hardware, viruses (particularly Covid), rocketry, and literature. During the discussion one error in a Wikipedia page was discussed and hopefully we can get that fixed.

I think that everyone who attended will be interested in more such meetings. Overall I think this is a reasonable start to the Hack Day meetings, when I previously ran such meetings they often ended up being more social events than serious hacking events and that’s OK too.

One conclusion that we came to regarding meetings is that they should always be well announced in email and that the iCal file isn’t useful for everyone. Discussion continues on the best methods of announcing meetings but I anticipate that better email will get more attendance.

,

Tim RileyOpen source status update, March 2022

My OSS work in March was a bit of a grind, but I made progress nonetheless. I worked mostly on relocating and refactoring the Hanami action and view integration code.

For some context, it was back in May 2020 that I first write the action/view integration code for Hanami 2.0. Back then, there were a couple of key motivators:

  • Reduce boilerplate to an absolute minimum, to the extent that simply inheriting from Hanami::View within a slice would give you a view class fully integrated with the Hanami application.
  • Locate the integration code in the non-core gems themselves (i.e. in the hanami-controller and hanami-view gems, rather than hanami), to help set an example for how alternative implementations may also integrate with the framework.

Since then, we’ve learnt a few things:

  • As we’ve gone about refining the core framework, we’ve wound up having to synchronize changes from time to time across the hanami, hanami-controller, and hanami-view gems all at once.
  • Other Hanami contributors have noted that the original integration approach was a little too “magical,” and didn’t allow users any path to opt out of the integration code.

Once I finished my work on the concrete slice classes last month, I decided that now was the time to address these concerns, to bring the action and view class integrations back into the hanami gem, and to take a different approach to activating the integration code.

The work in progress is over in this PR, and thankfully, it’s nearly done!

The impact within Hanami 2 applications will be fairly minimal: the biggest change is that your base action and view classes will now inherit from application variants:

# slices/main/lib/action/base.rb

require "hanami/application/action"

module Main
  module Action
    class Base < Hanami::Application::Action
      # Previously, this inherited from Hanami::Action
    end
  end
end

By using this explicit application superclass for actions and views, we hopefully make it easier for our users to understand and distinguish between the integrated and standalone variants of these classes. This distinct superclass should also provide us a clear place to hang extra API documentation relating to the integrated behavior of actions and views.

More importantly for the overall experience, Hanami::Application::Action and Hanami::Application::View are both now kept within the core hanami gem. While the framework heads into this final stretch of work before 2.0 final, this will allow us to keep together the aspects of the integration that tend to change together, giving us our best chance at providing a tested, reliable, streamlined actions and views experience.

This is a pragmatic move above all else — we’re a team with little time, so the more we can do to give ourselves confidence in this integrated experience working properly, like having all the code and tests together in one place, the quicker we should be able to get to the 2.0 release. Longer term, we’ll want to provide a first-class integration story for third party components, and I believe we can lead the way in how we deliver that via our actions and views, but that’s now firmly a post-2.0 concern in my mind.

In the meantime, I did take this opportunity to rethink and provide some better hooks for classes like Hanami::Application::View to integrate with the rest of the framework, chiefly via a new Hanami::SliceConfigurable module. You can see how it works by checking out the code for Hanami::Application::View itself:

# frozen_string_literal: true

require "hanami/view"
require_relative "../slice_configurable"
require_relative "view/slice_configured_view"

module Hanami
  class Application
    class View < Hanami::View
      extend Hanami::SliceConfigurable

      def self.configure_for_slice(slice)
        extend SliceConfiguredView.new(slice)
      end
    end
  end
end

Any class that extends Hanami::SliceConfigurable will have its own .configure_for_slice(slice) method called whenever it is sublcassed within a module namespace that happens to match the namespace managed by an Hanami slice. Using the slice object passed to that hook, that class can then read any slice- or application-level config to set itself up to integrate with the application.

In the example above, we extend a slice-specific instance of SliceConfiguredView, which will copy across application level view configs, as well configure the view’s part namespaces to match the slice’s namespace. The reason we build a module instance here (this module builder pattern is a whole technique that I’ll gladly go into one day, but it’s a little out of scope for these monthly updates) is so that we don’t have to keep any trace of the slice as state on the class after we’re done using it for configuration, making it so the resulting class is as standalone as possible, and not offering any way for its users to inadvertently couple themselves to the whole slice instance.

Overall, this change is feeling quite settled now. All the code has been moved in and refactored, and all that’s left is a final polishing pass before merge, which I hope I can get done this week! A huge thank you to Sean Collins for his original work in proposing an adjustment to our action integration code. It was Sean’s feedback and exploratory work here that got me off the fence here, and made it so easy to get started with these changes!

That’s it for me for now. See you all again next month, hopefully with some more continued core framework polishing.

,

Ian BrownIntroduction to GCVE

What is GCVE? Google Cloud VMware Engine, or GCVE, is a fully managed VMware hypervisor and associated management and networking components, (vSphere, NSX-T, vSAN and HCX) built on top of Google’s highly performant and scalable infrastructure with fully redundant and dedicated 100Gbps networking that provides 99.99% availability. The solution is integrated into Google Cloud Platform, so businesses benefit from having full access to GCP services, native VPC networking, Cloud VPN or Interconnect as well as all the normal security features you expect from GCP.

,

FLOSS Down Under - online free software meetingsMarch 2022 Meeting

Meeting Report

The March 2022 meeting went reasonably well. Everyone seemed to have fun and learn useful things about computers. After 2 hours my Internet connection dropped out which stopped the people who were using VMs from doing the tutorial. Fortunately most people seemed ready for a break so we ended the meeting. The early and abrupt ending of the meeting was a disappointment but it wasn’t too bad, the meeting would probably only have gone for another half hour otherwise.

The BigBlueButton system was shown to be effective for training when one person got confused with the Debian package configuration options for Postfix and they were able to share the window with everyone else to get advice. I was also confused by that stage.

Future Meetings

The main feature of the meeting was training in setting up a mailserver with Postfix, here are the lecture notes for it [1]. The consensus at the end of the meeting was that people wanted more of that for the April meeting. So for the April meeting I will add to the Postfix Training to include SpamAssassin, SPF, DKIM, and DMARC. For the start of the next meeting instead of providing bare Debian installations for the VMs I’ll provide a basic Postfix/Dovecot setup so people can get straight into SpamAssassin etc.

For the May meeting training on SE Linux was requested.

Social Media

Towards the end of the meeting we discussed Matrix and federated social media. LUV has a Matrix server and I can give accounts to anyone who’s involved in FOSS in the Australia and New Zealand area. For Mastodon the NZOSS Mastodon server [2] seems like a good option. I have an account there to try Mastodon, my Mastodon address is @etbe@mastodon.nzoss.nz .

We are going to make Matrix a primary communication method for the Flounder group, the room is #flounder:luv.asn.au . My Matrix address is @etbe:luv.asn.au .

,

FLOSS Down Under - online free software meetingsMailing List

We now have a mailing list see https://lists.linux.org.au/mailman/listinfo/flounder for information, the address to post to the list is flounder@lists.linux.org.au..

We also have a new URL for the blog and events. See the right sidebar for the link to the iCal file which can be connected to Google Calendar and most online calendaring systems.

,

FLOSS Down Under - online free software meetingsFirst Meeting Success

We just had the first Flounder meeting which went well. Had some interesting discussion of storage technology, I learnt a few new things. Some people did the ZFS training and BTRFS training and we had lots of interesting discussion.

Andrew Pam gave a summary of new things in Linux and talked about the sites lwn.net, gamingonlinux.com, and cnx-software.com that he uses to find Linux news. One thing he talked about is the latest developments with SteamDeck which is driving Linux support in Steam games. The site protondb.com tracks Linux support in Steam games.

We had some discussion of BPF, for an introduction to that technology see the BPF lecture from LCA 2022.

Next Meeting

The next meeting (Saturday 5th of March 1PM Melbourne time) will focus on running your own mail server which is always of interest to people who are interested in system administration and which is probably of more interest than usual because of Google forcing companies with “a legacy G Suite subscription” to transition to a more expensive “Business family” offering.

,

Stewart SmithAdventures in the Apple Partition Map (Part 2 of the continuing adventures with the Apple Power Macintosh 7200/120 PC Compatible)

I “recently” wrote about obtaining a new (to me, actually quite old) computer over in The Apple Power Macintosh 7200/120 PC Compatible (Part 1). This post is a bit of a detour, but may help others understand why some images they download from the internet don’t work.

Disk partitioning is (of course) a way to divide up a single disk into multiple volumes (partitions) for different uses. While the idea is similar, computer platforms over the ages have done this in a variety of different ways, with varying formats on disk, and varying limitations. The ones that you’re most likely to be familiar with are the MBR partitioning scheme (from the IBM PC), and the GPT partitioning scheme (common for UEFI systems such as the modern PC and Mac). One you’re less likely to be familiar with is the Apple Partition Map scheme.

The way all IBM PCs and compatibles worked from the introduction of MS-DOS 2.0 in 1983 until some time after 2005 was the Master Boot Record partitioning scheme. It was outrageously simple: of the first 512 byte sector of a disk, the first 446 bytes was for the bootstrapping code (the “boot sector”), the last 2 bytes were for the magic two bytes telling the BIOS this disk was bootable, and the other 64 bytes were four entries of 16 bytes, each describing a disk partition. The Wikipedia page is a good overview of what it all looks like. Since “four partitions should be enough for anybody” wasn’t going to last, DOS 3.2 introduced “extended partitions” which was just using one of those 4 partitions as another similar data structure that could point to more partitions.

In the 1980s (similar to today), the Macintosh was, of course, different. The Apple Partition Map is significantly more flexible than the MBR on PCs. For a start, you could have more than four partitions! You could actually have a lot more than four partitions, as the Apple Partition Map is a single 512-byte sector for each partition, and the partition map is itself a partition. Instead of being block 0 (like the MBR is), it actually starts at block 1, and is contiguous (The Driver Descriptor Record is what’s at block 0). So, once created, it’s hard to extend. Typically it’d be created as 64×512-byte entries, for 32kb… which turns out is actually about enough for anyone.

The Inside Macintosh reference on the SCSI Manager goes through more detail as to these structures. If you’re wondering what language all the coding examples are in, it’s Pascal – which was fairly popular for writing Macintosh applications in back in the day.

But the actual partition map isn’t the “interesting” part of all this (and yes, the quotation marks are significant here), because Macs are pretty darn finicky about what disks to boot off, which gets to be interesting if you’re trying to find a CD-ROM image on the internet from which to boot, and then use to install an Operating System from.

Stewart SmithEvery time I program a Mac…

… the preferred programming language changes.

I never programmed a 1980s Macintosh actually in the 1980s. It was sometime in the early 1990s that I first experienced Microsoft Basic for the Macintosh. I’d previously (unknowingly at the time as it was branded Commodore) experienced Microsoft BASIC on the Commodore 16, Commodore 64, and even the Apple ][, but the Macintosh version was something else. It let you do some pretty neat things such as construct a GUI with largely the same amount of effort as it took to construct a Text based UI on the micros I was familiar with.

Okay, to be fair, I’d also dabbled in Microsoft QBasic that came bundled with MS-DOS of the era, which let you do a whole bunch of graphics – so you could theoretically construct a GUI with it. Something I did attempt to do. Programming on the Mac was so much easier to construct a GUI.

Of course, Microsoft Basic wasn’t the preferred way to program on the Macintosh. At that time it was largely Pascal, with C being something that also existed – but you were going to see Pascal in Inside Macintosh. It was probably somewhat fortuitous that I’d poked at Pascal a bit as something alternate to look at in the high school computing classes. I can only remember using TurboPascal on DOS systems and never actually writing Pascal on the Macintosh.

By the middle part of the 1990s though, I was firmly incompetently writing C on the Mac. No doubt the quality of my code increased after I’d done some university courses actually covering the language rather than the only practical way I had to attempt to write anything useful being looking at Inside Macintosh examples in Pascal and “C for Dummies” which was very not-Macintosh. Writing C on UNIX/Linux was a lot easier – everything was made for it, including Actual Documentation!

Anyway, in the early 2000s I ran MacOS X for a bit on my white iBook G3, and did a (very) small amount of any GUI / Project Builder (the precursor to Xcode) related development – instead largely focusing on command line / X11 things. The latest coolness being to use Objective-C to program applications (unless you were bringing over your Classic MacOS Carbon based application, then you could still write C). Enter some (incompetent) Objective-C coding!

Then Apple went to x86, so the hardware ceased being interesting, and I had no reason to poke at it even as a side effect of having hardware that could run the software stack. Enter a long-ass time of Debian, Ubuntu, and Fedora on laptops.

Come 2022 though, and (for reasons I should really write up), I’m poking at a Mac again and it’s now Swift as the preferred way to write apps. So, I’m (incompetently) hacking away at Swift code. I have to admit, it’s pretty nice. I’ve managed to be somewhat productive in a relative short amount of time, and all the affordances in the language gear towards the kind of safety that is a PITA when coding in C.

So this is my WIP utility to be able to import photos from a Shotwell database into the macOS Photos app:

There’s a lot of rough edges and unknowns left, including how to actually do the import (it looks like there’s going to be Swift code doing AppleScript things as the PhotoKit API is inadequate). But hey, some incompetent hacking in not too much time has a kind-of photo browser thing going on that feels pretty snappy.

,

Robert Collinshyper combinators in Rust

Recently I read Michael Snoyman’s post on combining Axum, Hyper, Tonic and Tower. While his solution worked, it irked me – it seemed like there should be a much tighter solution possible.

I can deep dive into the code in a later post perhaps, but I think there are four points of difference. One, since the post was written Axum has started boxing its routes : so the enum dispatch approach taken, which delivers low overheads actually has no benefits today.

Two, while writing out the entire type by hand has some benefits, async code is much more pithy.

Thirdly, the code in the post is entirely generic, except the routing function itself.

And fourth, the outer Service<AddrStream> is an unnecessary layer to abstract over: given the similar constraints – the inner Service must take Request<..>, it is possible to just not use a couple of helpers and instead work directly with Service<Request...>.

So, onto a pithier version.

First, the app server code itself.

use std::{convert::Infallible, net::SocketAddr};

use axum::routing::get;
use hyper::{server::conn::AddrStream, service::make_service_fn};
use hyper::{Body, Request};
use tonic::async_trait;

use demo::echo_server::{Echo, EchoServer};
use demo::{EchoReply, EchoRequest};

struct MyEcho;

#[async_trait]
impl Echo for MyEcho {
    async fn echo(
        &self,
        request: tonic::Request<EchoRequest>,
    ) -> Result<tonic::Response<EchoReply>, tonic::Status> {
        Ok(tonic::Response::new(EchoReply {
            message: format!("Echoing back: {}", request.get_ref().message),
        }))
    }
}

#[tokio::main]
async fn main() {
    let addr = SocketAddr::from(([0, 0, 0, 0], 3000));

    let axum_service = axum::Router::new().route("/", get(|| async { "Hello world!" }));

    let grpc_service = tonic::transport::Server::builder()
        .add_service(EchoServer::new(MyEcho))
        .into_service();

    let both_service =
        demo_router::Router::new(axum_service, grpc_service, |req: &Request<Body>| {
            Ok::<bool, Infallible>(
                req.headers().get("content-type").map(|x| x.as_bytes())
                    == Some(b"application/grpc"),
            )
        });

    let make_service = make_service_fn(move |_conn: &AddrStream| {
        let both_service = both_service.clone();
        async { Ok::<_, Infallible>(both_service) }
    });

    let server = hyper::Server::bind(&addr).serve(make_service);

    if let Err(e) = server.await {
        eprintln!("server error: {}", e);
    }
}

Note the Router: it takes the two services and Fn to determine which to use on any given request. Then we just drop that composed service into make_service_fn and we’re done.

Next up we have the Router implementation. This is generic across any two Service<Request<...>> types as long as they are both Into<Bytes> for their Data, and Into<Box<dyn Error>> for errors.

use std::{future::Future, pin::Pin, task::Poll};

use http_body::combinators::UnsyncBoxBody;
use hyper::{body::HttpBody, Body, Request, Response};
use tower::Service;

#[derive(Clone)]
pub struct Router<First, Second, F> {
    first: First,
    second: Second,
    discriminator: F,
}

impl<First, Second, F> Router<First, Second, F> {
    pub fn new(first: First, second: Second, discriminator: F) -> Self {
        Self {
            first,
            second,
            discriminator,
        }
    }
}

impl<First, Second, FirstBody, FirstBodyError, SecondBody, SecondBodyError, F, FErr>
    Service<Request<Body>> for BinaryRouter<First, Second, F>
where
    First: Service<Request<Body>, Response = Response<FirstBody>>,
    First::Error: Into<Box<dyn std::error::Error + Send + Sync>> + 'static,
    First::Future: Send + 'static,
    First::Response: 'static,
    Second: Service<Request<Body>, Response = Response<SecondBody>>,
    Second::Error: Into<Box<dyn std::error::Error + Send + Sync>> + 'static,
    Second::Future: Send + 'static,
    Second::Response: 'static,
    F: Fn(&Request<Body>) -> Result<bool, FErr>,
    FErr: Into<Box<dyn std::error::Error + Send + Sync>> + Send + 'static,
    FirstBody: HttpBody<Error = FirstBodyError> + Send + 'static,
    FirstBody::Data: Into<bytes::Bytes>,
    FirstBodyError: Into<Box<dyn std::error::Error + Send + Sync>> + 'static,
    SecondBody: HttpBody<Error = SecondBodyError> + Send + 'static,
    SecondBody::Data: Into<bytes::Bytes>,
    SecondBodyError: Into<Box<dyn std::error::Error + Send + Sync>> + 'static,
{
    type Response = Response<
        UnsyncBoxBody<
            <hyper::Body as HttpBody>::Data,
            Box<dyn std::error::Error + Send + Sync + 'static>,
        >,
    >;
    type Error = Box<dyn std::error::Error + Send + Sync + 'static>;
    type Future =
        Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;

    fn poll_ready(
        &mut self,
        cx: &mut std::task::Context<'_>,
    ) -> std::task::Poll<Result<(), Self::Error>> {
        match self.first.poll_ready(cx) {
            Poll::Ready(Ok(())) => match self.second.poll_ready(cx) {
                Poll::Ready(Ok(())) => Poll::Ready(Ok(())),
                Poll::Ready(Err(e)) => Poll::Ready(Err(e.into())),
                Poll::Pending => Poll::Pending,
            },
            Poll::Ready(Err(e)) => Poll::Ready(Err(e.into())),
            Poll::Pending => Poll::Pending,
        }
    }

    fn call(&mut self, req: Request<Body>) -> Self::Future {
        let discriminant = { (self.discriminator)(&req) };
        let (first, second) = if matches!(discriminant, Ok(false)) {
            (Some(self.first.call(req)), None)
        } else if matches!(discriminant, Ok(true)) {
            (None, Some(self.second.call(req)))
        } else {
            (None, None)
        };
        let f = async {
            Ok(match discriminant.map_err(Into::into)? {
                true => second
                    .unwrap()
                    .await
                    .map_err(Into::into)?
                    .map(|b| b.map_data(Into::into).map_err(Into::into).boxed_unsync()),
                false => first
                    .unwrap()
                    .await
                    .map_err(Into::into)?
                    .map(|b| b.map_data(Into::into).map_err(Into::into).boxed_unsync()),
            })
        };
        Box::pin(f)
    }
}

Interesting things here – I use boxed_unsync to abstract over the body concrete type, and I implement the future using async code rather than as a separate struct. It becomes much smaller even after a few bits of extra type constraining.

One thing that flummoxed me for a little was the need to capture the future for the underlying response outside of the async block. Failing to do so provokes a 'static requirement which was tricky to debug. Fortunately there is a bug on making this easier to diagnose in rustc already. The underlying problem is that if you create the async block, and then dereference self, the type for impl of .first has to live an arbitrary time. Whereas by capturing the future immediately, only the impl of the future has to live an arbitrary time, and that doesn’t then require changing the signature of the function.

This is almost worth turning into a crate – I couldn’t see an existing one when I looked, though it does end up rather small – < 100 lines. What do you all think?

FLOSS Down Under - online free software meetingsFirst Meeting Agenda

The first meeting will start at 1PM Australian Eastern time (Melbourne/Sydney) which is +1100 on Saturday the 5th of February.

I will start the video chat an hour early in case someone makes a timezone mistake and gets there an hour before it starts. If anyone else joins early we will have random chat until the start time (deliberately avoiding topics worthy of the main meeting). The link http://b.coker.com.au will redirect to the meeting URL on the day.

The first scheduled talk is a summary and discussion of free software related news. Anyone who knows of something new that excites them is welcome to speak about it.

The main event is discussion of storage technology and hands-on training on BTRFS and ZFS for those who are interested. Here are the ZFS training notes and here are the BTRFS training notes. Feel free to do the training exercises on your own VM before the meeting if you wish.

Then discussion of the future of the group and the use of FOSS social media. While social media is never going to be compulsory some people will want to use it to communicate and we could run some servers for software that is considered good (lots of server capacity is available).

Finally we have to plan future meetings and decide on which communication methods are desired.

The BBB instance to be used for the video conference is sponsored by NZOSS and Catalyst Cloud.

,

OpenSTEMCovering the federal election, before the election

Since PM Scott Morrison did not announce the federal election date last week, it will now be held somewhere between March and May (see the post from ABC’s Antony Green for details). Various aspects of elections are covered in the Civics & Citizenship Australian Curriculum in Years 4, 5 and 6. Students are interested in […]

The post Covering the federal election, before the election first appeared on OpenSTEM Pty Ltd.

FLOSS Down Under - online free software meetingsFlounder Overview

Flounder is a new free software users group based in the Australia/NZ area. Flounder stands for FLOSS (Free Libre Open Source Software) down under.

Here is my blog post describing the initial idea, the comment from d3Xt3r suggested the name. Flounder is a group of fish that has species native to Australia and NZ.

The main aim is to provide educational benefits to free software users via an online meeting that can’t be obtained by watching YouTube videos etc in a scope that is larger than one country. When the pandemic ends we will keep running this as there are benefits to be obtained from a meeting of a wide geographic scope that can’t be obtained by meetings in a single city. People from other countries are welcome to attend but they aren’t the focus of the meeting.

Until we get a better DNS name the address http://b.coker.com.au will redirect to the BBB instance used for online meetings (the meeting address isn’t yet setup so it redirects to the blog). The aim is that there will always be a short URL for the meeting so anyone who has one device lose contact can quickly type the URL into their backup device.

The first meeting will be on the 5th of Feb 2022 at 1PM Melbourne time +1100. When we get a proper domain I’ll publish a URL for an iCal file with entries for all meetings. I will also find some suitable way for meeting times to be localised (I’m sure there’s a WordPress plugin for that).

For the hands-on part of the meetings there will be virtual machine images you can download to run on your own system (tested with KVM, should work with other VM systems) and the possibility of logging in to a running VM. The demonstration VMs will have public IPv6 addresses and will also be available through different ports on a single IPv4 address, having IPv6 on your workstation will be convenient for you but you can survive without it.

Linux Australia has a list of LUGs in Australia, is there a similar list for NZ? One thing I’d like to see is a list of links for iCal files for all the meetings and also an iCal aggregator that for all iCal feeds of online meetings. I’ll host it myself if necessary, but it’s probably best to do it via Linux Australia (Linux Australasia?) if possible.

,

Jan SchmidtPulling on a thread

I’m attending the https://linux.conf.au/ conference online this weekend, which is always a good opportunity for some sideline hacking.

I found something boneheaded doing that today.

There have been a few times while inventing the OpenHMD Rift driver where I’ve noticed something strange and followed the thread until it made sense. Sometimes that leads to improvements in the driver, sometimes not.

In this case, I wanted to generate a graph of how long the computer vision processing takes – from the moment each camera frame is captured until poses are generated for each device.

To do that, I have a some logging branches that output JSON events to log files and I write scripts to process those. I used that data and produced:

Pose recognition latency.
dt = interpose spacing, delay = frame to pose latency

Two things caught my eye in this graph. The first is the way the baseline latency (pink lines) increases from ~20ms to ~58ms. The 2nd is the quantisation effect, where pose latencies are clearly moving in discrete steps.

Neither of those should be happening.

Camera frames are being captured from the CV1 sensors every 19.2ms, and it takes that 17-18ms for them to be delivered across the USB. Depending on how many IR sources the cameras can see, figuring out the device poses can take a different amount of time, but the baseline should always hover around 17-18ms because the fast “device tracking locked” case take as little as 1ms.

Did you see me mention 19.2ms as the interframe period? Guess what the spacing on those quantisation levels are in the graph? I recognised it as implying that something in the processing is tied to frame timing when it should not be.

OpenHMD Rift CV1 tracking timing

This 2nd graph helped me pinpoint what exactly was going on. This graph is cut from the part of the session where the latency has jumped up. What it shows is a ~1 frame delay between when the frame is received (frame-arrival-finish-local-ts) before the initial analysis even starts!

That could imply that the analysis thread is just busy processing the previous frame and doesn’t get start working on the new one yet – but the graph says that fast analysis is typically done in 1-10ms at most. It should rarely be busy when the next frame arrives.

This is where I found the bone headed code – a rookie mistake I wrote when putting in place the image analysis threads early on in the driver development and never noticed.

There are 3 threads involved:

  • USB service thread, reading video frame packets and assembling pixels in framebuffers
  • Fast analysis thread, that checks tracking lock is still acquired
  • Long analysis thread, which does brute-force pose searching to reacquire / match unknown IR sources to device LEDs

These 3 threads communicate using frame worker queues passing frames between each other. Each analysis thread does this pseudocode:

while driver_running:
    Pop a frame from the queue
    Process the frame
    Sleep for new frame notification

The problem is in the 3rd line. If the driver is ever still processing the frame in line 2 when a new frame arrives – say because the computer got really busy – the thread sleeps anyway and won’t wake up until the next frame arrives. At that point, there’ll be 2 frames in the queue, but it only still processes one – so the analysis gains a 1 frame latency from that point on. If it happens a second time, it gets later by another frame! Any further and it starts reclaiming frames from the queues to keep the video capture thread fed – but it only reclaims one frame at a time, so the latency remains!

The fix is simple:

while driver_running:
   Pop a frame
   Process the frame
   if queue_is_empty():
     sleep for new frame notification

Doing that for both the fast and long analysis threads changed the profile of the pose latency graph completely.

Pose latency and inter-pose spacing after fix

This is a massive win! To be clear, this has been causing problems in the driver for at least 18 months but was never obvious from the logs alone. A single good graph is worth a thousand logs.

What does this mean in practice?

The way the fusion filter I’ve built works, in between pose updates from the cameras, the position and orientation of each device are predicted / updated using the accelerometer and gyro readings. Particularly for position, using the IMU for prediction drifts fairly quickly. The longer the driver spends ‘coasting’ on the IMU, the less accurate the position tracking is. So, the sooner the driver can get a correction from the camera to the fusion filter the less drift we’ll get – especially under fast motion. Particularly for the hand controllers that get waved around.

Before: Left Controller pose delays by sensor
After: Left Controller pose delays by sensor

Poses are now being updated up to 40ms earlier and the baseline is consistent with the USB transfer delay.

You can also visibly see the effect of the JPEG decoding support I added over Christmas. The ‘red’ camera is directly connected to USB3, while the ‘khaki’ camera is feeding JPEG frames over USB2 that then need to be decoded, adding a few ms delay.

The latency reduction is nicely visible in the pose graphs, where the ‘drop shadow’ effect of pose updates tailing fusion predictions largely disappears and there are fewer large gaps in the pose observations when long analysis happens (visible as straight lines jumping from point to point in the trace):

Before: Left Controller poses
After: Left Controller poses

,

Colin CharlesThis thing is still on?

Yes, the blog is still on. January 2004 I moved to WordPress, and it is still here January 2022. I didn’t write much last year (neither here, not experimenting with the Hey blog). I didn’t post anything to Instagram last year either from what I can tell, just a lot of stories.

August 16 2021, I realised I was 1,000 days till May 12 2024, which is when I become 40. As of today, that leads 850 days. Did I squander the last 150 days? I’m back to writing almost daily in the Hobonichi Techo (I think last year and the year before were mostly washouts; I barely scribbled anything offline).

I got a new Apple Watch Series 7 yesterday. I can say I used the Series 4 well (79% battery life), purchased in the UK when I broke my Series 0 in Edinburgh airport.

TripIt stats for last year claimed 95 days on the road. This is of course, a massive joke, but I’m glad I did get to visit London, Lisbon, New York, San Francisco, Los Angeles without issue. I spent a lot of time in Kuantan, a bunch of Langkawi trips, and also, I stayed for many months at the Grand Hyatt Kuala Lumpur during the May lockdowns (I practically stayed there all lockdown).

With 850 days to go till I’m 40, I have plenty I would like to achieve. I think I’ll write a lot more here. And elsewhere. Get back into the habit of doing. And publishing by learning and doing. No fear. Not that I wasn’t doing, but its time to be prolific with what’s been going on.

The post This thing is still on? first appeared on Colin Charles Agenda.

,

,

,

Gary PendergastWordPress and web3

Blockchain. Cryptocurrency. Ethereum. NFTs. DAOs. Smart Contracts. web3. It’s impossible to avoid the blockchain hype machine these days, but it’s often just as difficult to decipher what it all means.

On top of that, discourse around web3 is extremely polarising: everyone involved is very keen to a) pick a team, and b) get you to join their team. If you haven’t picked a team, you must be secretly with the other team.

Max Read made a compelling argument that the web3 debate is in fact two different debates:

But, OK, what is the root disagreement, exactly? The way I read it there are two broad “is web3 bullshit?” debates, not just one, centered around the following questions:

Can the blockchain do anything that other currently existing technology cannot do and/or do anything better or more efficiently than other currently existing technology?

Will the blockchain form the architecture of the internet of the future (i.e. “web3”), and/or will blockchain-native companies and organizations become important and powerful?

Max Read — Is web3 bullshit?

I’m inclined to agree with Max’s analysis here: there’s a technical question, and there’s a business/cultural question. It’s hard to separate the two when every day sees new headlines about millions of dollars being stolen or scammed; or thousands of people putting millions of dollars into highly optimistic ventures. There are extreme positives and extreme negatives happening all the time in the web3 world.

With that in mind, I want to take a step back from the day-to-day excitement of cryptocurrency and web3, and look at some of the driving philosophies espoused by the movement.

Philosophies of web3

There are a lot of differing viewpoints on web3, every individual has a slightly different take on it. There are three broad themes that stand out, however.

Decentralised

Blockchain-based technology is inherently distributed (with some esoteric caveats, but we can safely ignore them for now). In a world where the web centres around a handful of major services, where we’ve seen the harm that the likes of Facebook and YouTube can inflict on society, it’s not surprising that decentralisation would be a powerful theme drawing in anyone looking for an alternative.

Decentralisation isn’t new to the Internet, of course: it’s right there in the name. This giant set of “interconnected networks” has been decentralised from the very beginning. It’s not perfect, of course: oppressive governments can take control of the borders of their portion of the Internet, and we’ve come to rely on a handful of web services to handle the trickier parts of using the web. But fundamentally, that decentralised architecture is still there. I can still set up a web site hosted on my home computer, which anyone in the world could access.

I don’t do that, however, for the same reason that web3 isn’t immune from centralised services: Centralisation is convenient. Just as we have Facebook, or Google, or Amazon as giant centralised services on the current web, we can already see similar services appearing for web3. For payments, Coinbase has established itself as a hugely popular place exchange cryptocurrencies and traditional currencies. For NFTs, OpenSea is the service where you’ll find nearly every NFT collection. MetaMask keeps all of your crypto-based keys, tokens, and logins in a single “crypto wallet”.

Centralisation is convenient.

While web3 proponents give a lot of credence to the decentralised nature of cryptocurrency being a driver of popularity, I’m not so sure. At best, I’m inclined to think that decentralisation is table stakes these days: you can’t even get started as a global movement without a strong commitment to decentralisation.

But if decentralisation isn’t the key, what is?

Ownership

When we talk about ownership in web3, NFTs are clearly the flavour of the month, but recent research indicates that the entire NFT market is massively artificially inflated.

Rather than taking pot-shots at the NFT straw man, I think it’s more interesting to look at the idea of ownership in terms of attribution. The more powerful element of this philosophy isn’t about who owns something, it’s who created it. NFTs do something rather novel with attribution, allowing royalty payments to the original artist every time an NFT is resold. I love this aspect: royalties shouldn’t just be for movie stars, they should be for everyone.

Comparing that to the current web, take the 3 paragraphs written by Max Read that I quoted above. I was certainly under no technical obligation to show that it was a quote, to attribute it to him, or to link to the source. In fact, it would have been easier for me to just paste his words into this post, and pretend they were my own. I didn’t, of course, because I feel an ethical obligation to properly attribute the quote.

In a world where unethical actors will automatically copy/paste your content for SEO juice (indeed, I expect this blog post to show up on a bunch of these kinds of sites); where massive corporations will consume everything they can find about you, in order to advertise more effectively to you, it’s not at all surprising that people are looking for a technical solution for taking back control of their data, and for being properly attributed for their creations.

The interesting element of this philosophy isn’t about who owns something, it’s who created it.

That’s not to say that existing services discourage attribution: a core function of Twitter is retweets, a core function of Tumblr is reblogging. WordPress still supports trackbacks, even if many folks turn them off these days.

These are all blunt instruments, though, aimed at attributing an entire piece, rather than a more targeted approach. What I’d really like is a way to easily quote and attribute a small chunk of a post: 3 paragraphs (or blocks, if you want to see where I’m heading 😉), inserted into my post, linking back to where I got them from. If someone chooses to quote some of this post, I’d love to receive a pingback just for that quote, so it can be seen in the right context.

The functionality provide by Twitter and Tumblr is less of a technologically-based enforcement of attribution, and more of an example of paving the cow path: by and large, people want to properly attribute others, providing the tools to do so can easily become a fundamental part of how any software is used.

These tools only work so long as there’s an incentive to use them, however. web3 certainly provides the tools to attribute others, but much like SEO scammers copy/pasting blog posts, the economics of the NFT bubble is clearly a huge incentive to ignore those tools and ethical obligations, to the point that existing services have had to build additional features just to detect this abuse.

Monetisation

With every major blockchain also being a cryptocurrency, monetisation is at the heart of the entire web3 movement. Every level of the web3 tech stack involves a cryptocurrency-based protocol. This naturally permeates through the entire web3 ecosystem, where money becomes a major driving factor for every web3-based project.

And so, it’s impossible to look at web3 applications without also considering the financial aspect. When you have to pay just to participate, you have to ask whether every piece of content you create is “worth it”.

Again, let’s go back to the 3 paragraphs I quote above. In a theoretical web3 world, I’d publish this post on a blockchain in some form or another, and that act would also likely include noting that I’d quoted 3 blocks of text attributed to Max Read. I’d potentially pay some amount of money to Max, along with the fees that every blockchain charges in order to perform a transaction. While this process is potentially helpful to the original author at a first glance, I suspect the second and third order effects will be problematic. Having only just clicked the Publish button a few seconds earlier, I’m already some indeterminate amount of money out of pocket. Which brings me back to the question, is this post “worth it”? Will enough people tip/quote/remix/whatever me, to cover the cost of publishing? When every creative work must be viewed through a lens of financial impact, it fundamentally alters that creative process.

When you have to pay just to participate, you have to ask whether every piece of content you create is “worth it”.

Ultimately, we live in a capitalist society, and everyone deserves the opportunity to profit off their work. But by baking monetisation into the underlying infrastructure of web3, it becomes impossible to opt-out. You either have the money to participate without being concerned about the cost, or you’re going to need to weigh up every interaction by whether or not you can afford it.

Web3 Philosophies in WordPress

After breaking it all down, we can see that it’s not all black-and-white. There are some positive parts of web3, and some negative parts. Not that different to the web of today, in fact. 🙂 That’s not to say that either approach is the correct one: instead, we should be looking to learn from both, and produce something better.

Decentralised

I’ve long been a proponent of leveraging the massive install base of WordPress to provide distributed services to anyone. Years ago, I spoke about an idea called “Connected WordPress” that would do exactly that. While the idea didn’t gain a huge amount of traction at the time, the DNA of the Connected WordPress concept shares a lot of similar traits to the decentralised nature of web3.

I’m a big fan of decentralised technologies as a way for individuals to claw back power over their own data from the governments and massive corporations that would prefer to keep it all centralised, and I absolutely think we should be exploring ways to make the existing web more resistant to censorship.

At the same time, we have to acknowledge that there are certainly benefits to centralisation. As long as people have the freedom to choose how and where they participate, and centralised services are required to play nicely with self hosted sites, is there a practical difference?

I quite like how Solid allows you have it both ways, whilst maintaining control over your own data.

Ownership Attribution

Here’s the thing about attribution: you can’t enforce it with technology alone. Snapchat have indirectly demonstrated exactly this problem: in order to not lose a message, people would screenshot or record the message on their phone. In response, Snapchat implemented a feature to notify the other party when you screenshot a message from them. To avoid this, people will now use a second phone to take a photo or video of the message. While this example isn’t specifically about attribution, it demonstrates the problem that there’s no way to technologically restrict how someone interacts with content that you’ve published, once they’ve been granted access.

Instead of worrying about technical restrictions, then, we should be looking at how attribution can be made easier.

IndieWeb is a great example of how this can be done in a totally decentralised fashion.

Monetisation

I’m firmly of the opinion that monetisation of the things you create should be opt-in, rather than opt-out.

Modern society is currently obsessed with monetising everything, however. It comes in many different forms: hustle culture, side gigs, transforming hobbies into businesses, meme stocks, and cryptocurrencies: they’re all symptoms of this obsession.

I would argue that, rather than accepting as fait accompli that the next iteration of the web will be monetised to the core, we should be pushing back against this approach. Fundamentally, we should be looking to build for a post scarcity society, rather than trying to introduce scarcity where there previously was none.

While we work towards that future, we should certainly be easier for folks to monetise their work, but the current raft of cryptocurrencies just aren’t up to the task of operating as… currencies.

What Should You Do?

Well, that depends on what your priorities are. The conversations around web3 are taking up a lot of air right now, so it’s possible to get the impression web3 will be imminently replacing everything. It’s important to keep perspective on this, though. While there’s a lot of money in the web3 ecosystem right now, it’s dwarfed by the sheer size of the existing web.

If you’re excited about the hot new tech, and feeling inspired by the ideas espoused in web3 circles? Jump right in! I’m certain you’ll find something interesting to work on.

Always wanted to get into currency speculation, but didn’t want to deal with all those pesky “regulations” and “safeguards”? Boy howdy, are cryptocurrencies or NFTs the place for you. (Please don’t pretend that this paragraph is investment advice, it is nothing of the sort.)

Want to continue building stuff on the web, and you’re willing to learn new things when you need them, but are otherwise happy with your trajectory? Just keep on doing what you’re doing. Even if web3 does manage to live up to the hype, it’ll take a long time for it to be adopted by the mainstream. You’ll have years to adapt.

Final Thoughts

There are some big promises associated with web3, many of which sound very similar to the promises that were made around web 2.0, particularly around open APIs, and global interoperability. We saw what happened when those kinds of tools go wrong, and web3 doesn’t really solve those problems. It may exacerbate them in some ways, since it’s impossible to delete your data from a blockchain.

That said, (and I say this as a WordPress Core developer), just because a particular piece of software is not the optimal technical solution doesn’t mean it won’t become the most popular. Market forces can be a far stronger factor that technical superiority. There are many legitimate complaints about blockchain (including performance, bloat, fit for purpose, and security) that have been levelled against WordPress in the past, but WordPress certainly isn’t slowing down. I’m not even close to convinced that blockchain is the right technology to base the web on, but I’ve been doing this for too long to bet everything against it.

Markets can remain irrational a lot longer than you and I can remain solvent.

—A. Gary Shilling

As for me, well… 😄

I remain sceptical of web3 as it’s currently defined, but I think there’s room to change it, and to adopt the best bits into the existing web. Web 1.0 didn’t magically disappear when Web 2.0 rolled in, it adapted. Maybe we’ll look back in 10 years and say this was a time when the web fundamentally changed. Or, maybe we’ll refer to blockchain in the same breath as pets.com, and other examples from the dotcom boom of the 1990’s.

The Net interprets censorship as damage and routes around it.

—John Gilmore

This quote was originally referring to Usenet, but it’s stayed highly relevant in the decades since. I think it applies here, too: if the artificial scarcity built into web3 behaves too much like censorship, preventing people from sharing what they want to share, the internet (or, more accurately, the billions of people who interact with the internet) will just… go around it. It won’t all be smooth sailing, but we’ll continue to experiment, evolve, and adapt as it changes.

Personally, I think now is a great time for us to be embracing the values and ideals of projects like Solid, and IndieWeb. Before web3 referred to blockchains, it was more commonly used in reference to the Semantic Web, which is far more in line with WordPress’ ideals, whilst also matching many of the values prioritised by the new web3. As a major driver of the Open Web, WordPress can help people own their content in a sustainable way, engage with others on their own terms, and build communities that don’t depend on massive corporations or hand-wavy magical tech solutions.

Don’t get too caught up in the drama of whatever is the flavour of the month. I’m optimistic about the long term resilience of the internet, and I think you should be, too. 🥳

,

Jan Schmidt2.5 years of Oculus Rift

Once again time has passed, and another update on Oculus Rift support feels due! As always, it feels like I’ve been busy with work and not found enough time for Rift CV1 hacking. Nevertheless, looking back over the history since I last wrote, there’s quite a lot to tell!

In general, the controller tracking is now really good most of the time. Like, wildly-swing-your-arms-and-not-lose-track levels (most of the time). The problems I’m hunting now are intermittent and hard to identify in the moment while using the headset – hence my enthusiasm over the last updates for implementing stream recording and a simulation setup. I’ll get back to that.

Outlier Detection

Since I last wrote, the tracking improvements have mostly come from identifying and rejecting incorrect measurements. That is, if I have 2 sensors active and 1 sensor says the left controller is in one place, but the 2nd sensor says it’s somewhere else, we’ll reject one of those – choosing the pose that best matches what we already know about the controller. The last known position, the gravity direction the IMU is detecting, and the last known orientation. The tracker will now also reject observations for a time if (for example) the reported orientation is outside the range we expect. The IMU gyroscope can track the orientation of a device for quite a while, so can be relied on to identify strong pose priors once we’ve integrated a few camera observations to get the yaw correct.

It works really well, but I think improving this area is still where most future refinements will come. That and avoiding incorrect pose extractions in the first place.

Plot of headset tracking – orientation and position

The above plot is a sample of headset tracking, showing the extracted poses from the computer vision vs the pose priors / tracking from the Kalman filter. As you can see, there are excursions in both position and orientation detected from the video, but these are largely ignored by the filter, producing a steadier result.

Left Touch controller tracking – orientation and position

This plot shows the left controller being tracked during a Beat Saber session. The controller tracking plot is quite different, because controllers move a lot more than the headset, and have fewer LEDs to track against. There are larger gaps here in the timeline while the vision re-acquires the device – and in those gaps you can see the Kalman filter interpolating using IMU input only (sometimes well, sometimes less so).

Improved Pose Priors

Another nice thing I did is changes in the way the search for a tracked device is made in a video frame. Before starting looking for a particular device it always now gets the latest estimate of the previous device position from the fusion filter. Previously, it would use the estimate of the device pose as it was when the camera exposure happened – but between then and the moment we start analysis more IMU observations and other camera observations might arrive and be integrated into the filter, which will have updated the estimate of where the device was in the frame.

This is the bit where I think the Kalman filter is particularly clever: Estimates of the device position at an earlier or later exposure can improve and refine the filter’s estimate of where the device was when the camera captured the frame we’re currently analysing! So clever. That mechanism (lagged state tracking) is what allows the filter to integrate past tracking observations once the analysis is done – so even if the video frame search take 150ms (for example), it will correct the filter’s estimate of where the device was 150ms in the past, which ripples through and corrects the estimate of where the device is now.

LED visibility model

To improve the identification of devices better, I measured the actual angle from which LEDs are visible (about 75 degrees off axis) and measured the size. The pose matching now has a better idea of which LEDs should be visible for a proposed orientation and what pixel size we expect them to have at a particular distance.

Better Smoothing

I fixed a bug in the output pose smoothing filter where it would glitch as you turned completely around and crossed the point where the angle jumps from +pi to -pi or vice versa.

Improved Display Distortion Correction

I got a wide-angle hi-res webcam and took photos of a checkerboard pattern through the lens of my headset, then used OpenCV and panotools to calculate new distortion and chromatic aberration parameters for the display. For me, this has greatly improved. I’m waiting to hear if that’s true for everyone, or if I’ve just fixed it for my headset.

Persistent Config Cache

Config blocks! A long time ago, I prototyped code to create a persistent OpenHMD configuration file store in ~/.config/openhmd. The rift-kalman-filter branch now uses that to store the configuration blocks that it reads from the controllers. The first time a controller is seen, it will load the JSON calibration block as before, but it will now store it in that directory – removing a multiple second radio read process on every subsequent startup.

Persistent Room Configuration

To go along with that, I have an experimental rift-room-config branch that creates a rift-room-config.json file and stores the camera positions after the first startup. I haven’t pushed that to the rift-kalman-filter branch yet, because I’m a bit worried it’ll cause surprising problems for people. If the initial estimate of the headset pose is wrong, the code will back-project the wrong positions for the cameras, which will get written to the file and cause every subsequent run of OpenHMD to generate bad tracking until the file is removed. The goal is to have a loop that monitors whether the camera positions seem stable based on the tracking reports, and to use averaging and resetting to correct them if not – or at least to warn the user that they should re-run some (non-existent) setup utility.

Video Capture + Processing

The final big ticket item was a rewrite of how the USB video frame capture thread collects pixels and passes them to the analysis threads. This now does less work in the USB thread, so misses fewer frames, and also I made it so that every frame is now searched for LEDs and blob identities tracked with motion vectors, even when no further analysis will be done on that frame. That means that when we’re running late, it better preserves LED blob identities until the analysis threads can catch up – increasing the chances of having known LEDs to directly find device positions and avoid searching. This rewrite also opened up a path to easily support JPEG decode – which is needed to support Rift Sensors connected on USB 2.0 ports.

Session Simulator

I mentioned the recording simulator continues to progress. Since the tracking problems are now getting really tricky to figure out, this tool is becoming increasingly important. So far, I have code in OpenHMD to record all video and tracking data to a .mkv file. Then, there’s a simulator tool that loads those recordings. Currently it is capable of extracting the data back out of the recording, parsing the JSON and decoding the video, and presenting it to a partially implemented simulator that then runs the same blob analysis and tracking OpenHMD does. The end goal is a Godot based visualiser for this simulation, and to be able to step back and forth through time examining what happened at critical moments so I can improve the tracking for those situations.

To make recordings, there’s the rift-debug-gstreamer-record branch of OpenHMD. If you have GStreamer and the right plugins (gst-plugins-good) installed, and you set env vars like this, each run of OpenHMD will generate a recording in the target directory (make sure the target dir exists):

export OHMD_TRACE_DIR=/home/user/openhmd-traces/
export OHMD_FULL_RECORDING=1

Up Next

The next things that are calling to me are to improve the room configuration estimation and storage as mentioned above – to detect when the poses a camera is reporting don’t make sense because it’s been bumped or moved.

I’d also like to add back in tracking of the LEDS on the back of the headset headband, to support 360 tracking. I disabled those because they cause me trouble – the headband is adjustable relative to the headset, so the LEDs don’t appear where the 3D model says they should be and that causes jitter and pose mismatches. They need special handling.

One last thing I’m finding exciting is a new person taking an interest in Rift S and starting to look at inside-out tracking for that. That’s just happened in the last few days, so not much to report yet – but I’ll be happy to have someone looking at that while I’m still busy over here in CV1 land!

As always, if you have any questions, comments or testing feedback – hit me up at thaytan@noraisin.net or on @thaytan Twitter/IRC.

Thank you to the kind people signed up as Github Sponsors for this project!

,

Matt PalmerDiscovering AWS IAM accounts

Let’s say you’re someone who happens to discover an AWS account number, and would like to take a stab at guessing what IAM users might be valid in that account. Tricky problem, right? Not with this One Weird Trick!

In your own AWS account, create a KMS key and try to reference an ARN representing an IAM user in the other account as the principal. If the policy is accepted by PutKeyPolicy, then that IAM account exists, and if the error says “Policy contains a statement with one or more invalid principals” then the user doesn’t exist.

As an example, say you want to guess at IAM users in AWS account 111111111111. Then make sure this statement is in your key policy:

{
  "Sid": "Test existence of user",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111111111111:user/bob"
  },
  "Action": "kms:DescribeKey",
  "Resource": "*"
}

If that policy is accepted, then the account has an IAM user named bob. Otherwise, the user doesn’t exist. Scripting this is left as an exercise for the reader.

Sadly, wildcards aren’t accepted in the username portion of the ARN, otherwise you could do some funky searching with ...:user/a*, ...:user/b*, etc. You can’t have everything; where would you put it all?

I did mention this to AWS as an account enumeration risk. They’re of the opinion that it’s a good thing you can know what users exist in random other AWS accounts. I guess that means this is a technique you can put in your toolbox safe in the knowledge it’ll work forever.

Given this is intended behaviour, I assume you don’t need to use a key policy for this, but that’s where I stumbled over it. Also, you can probably use it to enumerate roles and anything else that can be a principal, but since I don’t see as much use for that, I didn’t bother exploring it.

There you are, then. If you ever need to guess at IAM users in another AWS account, now you can!

,

Glen TurnerThe tyranny of product names

For a long time computer manufacturers have tried to differentiate themselves and their products from their competitors with fancy names with odd capitalisation and spelling. But as an author, using these names does a disservice to the reader: how are they to know that DEC is pronounced as if it was written Dec ("deck").

It's time we pushed back, and wrote for our readers, not for corporations.

It's time to use standard English rules for these Corporate Fancy Names. Proper names begin with a capital, unlike "ciscoSystems®" (so bad that Cisco itself moved away from it). Words are separated by spaces, so "Cisco Systems". Abbreviations and acronyms are written in lower case if they are pronounced as a word, in upper case if each letter is pronounced: so "ram" and "IBM®".

So from here on in I'll be using the following:

  • Face Book. Formerly, "Facebook®".
  • Junos. Formerly JUNOS®.
  • ram. Formerly RAM.
  • Pan OS. Formerly PAN-OS®.
  • Unix. Formerly UNIX®.

I'd encourage you to try this in your own writing. It does look odd for the first time, but the result is undeniably more readable. If we are not writing to be understood by our audience then we are nothing more than an unpaid member of some corporation's marketing team.



comment count unavailable comments

,

Dave HallYour Terraform Module Needs an Opinion

Learn why your Terraform modules should be opinionated.

,

Chris NeugebauerTalk Notes: On The Use and Misuse of Decorators

I gave the talk On The Use and Misuse of Decorators as part of PyConline AU 2021, the second in annoyingly long sequence of not-in-person PyCon AU events. Here’s some code samples that you might be interested in:

Simple @property implementation

This shows a demo of @property-style getters. Setters are left as an exercise :)


def demo_property(f):
    f.is_a_property = True
    return f


class HasProperties:

    def __getattribute__(self, name):
        ret = super().__getattribute__(name)
        if hasattr(ret, "is_a_property"):
            return ret()
        else:
            return ret

class Demo(HasProperties):

    @demo_property
    def is_a_property(self):
        return "I'm a property"

    def is_a_function(self):
        return "I'm a function"


a = Demo()
print(a.is_a_function())
print(a.is_a_property)

@run (The Scoped Block)

@run is a decorator that will run the body of the decorated function, and then store the result of that function in place of the function’s name. It makes it easier to assign the results of complex statements to a variable, and get the advantages of functions having less leaky scopes than if or loop blocks.

def run(f):
    return f()

@run
def hello_world():
    return "Hello, World!"

print(hello_world)

@apply (Multi-line stream transformers)

def apply(transformer, iterable_):

    def _applicator(f):

        return(transformer(f, iterable_))

    return _applicator

@apply(map, range(100)
def fizzbuzzed(i):
    if i % 3 == 0 and i % 5 == 0:
        return "fizzbuzz"
    if i % 3 == 0:
        return "fizz"
    elif i % 5 == 0:
        return "buzz"
    else:
        return str(i)

Builders


def html(f):
    builder = HtmlNodeBuilder("html")
    f(builder)
    return builder.build()


class HtmlNodeBuilder:
    def __init__(self, tag_name):
       self.tag_name = tag_name
       self.nodes = []

   def node(self, f):
        builder = HtmlNodeBuilder(f.__name__)
        f(builder)
        self.nodes.append(builder.build())

    def text(self, text):
        self.nodes.append(text)

    def build(self):
      nodes = "\n".join(self.nodes)
       return f"<{self.tag_name}>\n{nodes}\n</{self.tag_name}>"


@html
def document(b):
   @b.node
   def head(b):
       @b.node
       def title(b):
           b.text("Hello, World!")

   @b.node
   def body(b):
       for i in range(10, 0, -1):
           @b.node
           def p(b):
               b.text(f"{i}")

Code Registries

This is an incomplete implementation of a code registry for handling simple text processing tasks:

```python

def register(self, input, output):

def _register_code(f):
    self.registry[(input, output)] = f
    return f

return _register_code

in_type = (iterable[str], (WILDCARD, ) out_type = (Counter, (WILDCARD, frequency))

@registry.register(in_type, out_type) def count_strings(strings):

return Counter(strings)

@registry.register( (iterable[str], (WILDCARD, )), (iterable[str], (WILDCARD, lowercase)) ) def words_to_lowercase(words): …

@registry.register( (iterable[str], (WILDCARD, )), (iterable[str], (WILDCARD, no_punctuation)) ) def words_without_punctuation(words): …

def find_steps( self, input_type, input_attrs, output_type, output_attrs ):

hand_wave()

def give_me(self, input, output_type, output_attrs):

steps = self.find_steps(
    type(input), (), output_type, output_attrs
)

temp = input
for step in steps:
    temp = step(temp)

return temp

,

Jan SchmidtOpenHMD update

A while ago, I wrote a post about how to build and test my Oculus CV1 tracking code in SteamVR using the SteamVR-OpenHMD driver. I have updated those instructions and moved them to https://noraisin.net/diary/?page_id=1048 – so use those if you’d like to try things out.

The pandemic continues to sap my time for OpenHMD improvements. Since my last post, I have been working on various refinements. The biggest visible improvements are:

  • Adding velocity and acceleration API to OpenHMD.
  • Rewriting the pose transformation code that maps from the IMU-centric tracking space to the device pose needed by SteamVR / apps.

Adding velocity and acceleration reporting is needed in VR apps that support throwing things. It means that throwing objects and using gravity-grab to fetch objects works in Half-Life: Alyx, making it playable now.

The rewrite to the pose transformation code fixed problems where the rotation of controller models in VR didn’t match the rotation applied in the real world. Controllers would appear attached to the wrong part of the hand, and rotate around the wrong axis. Movements feel more natural now.

Ongoing work – record and replay

My focus going forward is on fixing glitches that are caused by tracking losses or outliers. Those problems happen when the computer vision code either fails to match what the cameras see to the device LED models, or when it matches incorrectly.

Tracking failure leads to the headset view or controllers ‘flying away’ suddenly. Incorrect matching leads to controllers jumping and jittering to the wrong pose, or swapping hands. Either condition is very annoying.

Unfortunately, as the tracking has improved the remaining problems get harder to understand and there is less low-hanging fruit for improvement. Further, when the computer vision runs at 52Hz, it’s impossible to diagnose the reasons for a glitch in real time.

I’ve built a branch of OpenHMD that uses GStreamer to record the CV1 camera video, plus IMU and tracking logs into a video file.

To go with those recordings, I’ve been working on a replay and simulation tool, that uses the Godot game engine to visualise the tracking session. The goal is to show, frame-by-frame, where OpenHMD thought the cameras, headset and controllers were at each point in the session, and to be able to step back and forth through the recording.

Right now, I’m working on the simulation portion of the replay, that will use the tracking logs to recreate all the poses.

Ian BrownNGINX Ingress Controller in GKE

GKE in Production - Part 2 This tutorial is part of a series I am creating on creating, running and managing Kubernetes on GCP the way I do in my day job. In this episode, we are covering how to setup a nginx ingress controller to handle incoming requests. Note: There may be some things I have skimmed over, if so or you see a glaring hole in my configuration, please drop me a line via the contact page linked at the top of the site.

,

Robert CollinsA moment of history

I’ve been asked more than once what it was like at the beginning of Ubuntu, before it was a company, when an email from someone I’d never heard of came into my mailbox.

We’re coming up on 20 years now since Ubuntu was founded, and I had cause to do some spelunking into IMAP archives recently… while there I took the opportunity to grab the very first email I received.

The Ubuntu long shot succeeded wildly. Of course, we liked to joke about how spammy those emails where: cold-calling a raft of Debian developers with job offers, some of them were closer to phishing attacks :). This very early one – I was the second employee (though I started at 4 days a week to transition my clients gradually) – was less so.

I think its interesting though to note how explicit a gamble this was framed as: a time limited experiment, funded for a year. As the company scaled this very rapidly became a hiring problem and the horizon had to be pushed out to 2 years to get folk to join.

And of course, while we started with arch in earnest, we rapidly hit significant usability problems, some of which were solvable with porcelain and shallow non-architectural changes, and we built initially patches, and then the bazaar VCS project to tackle those. But others were not: for instance, I recall exceeding the 32K hard link limit on ext3 due to a single long history during a VCS conversion. The sum of these challenges led us to create the bzr project, a ground up rethink of our version control needs, architecture, implementation and user-experience. While ultimately git has conquered all, bzr had – still has in fact – extremely loyal advocates, due to its laser sharp focus on usability.

Anyhow, here it is: one of the original no-name-here-yet, aka Ubuntu, introductory emails (with permission from Mark, of course). When I clicked through to the website Mark provided there was a link there to a fantastical website about a space tourist… not what I had expected to be reading in Adelaide during LCA 2004.


From: Mark Shuttleworth <xxx@xxx>
To: Robert Collins <xxx@xxx>
Date: Thu, 15 Jan 2004, 04:30

Tom Lord gave me your email address, I believe he’s
already sent you the email that I sent him so I’m sure
you have some background.

In short, I am going to fund some open source
development for a year. This is part of a new project
that I will be getting off the ground in the coming
weeks. I don’t know where it will lead, it’s flying in
the face of a stiff breeze but I think at the end of
the day it will at least fund a few very good open
source developers for a full year to work on the
projects they like most.

One of the pieces of the puzzle is high end source
code management. I’ll be looking to build an
infrastructure that will manage source code for
between 100 and 8000 open source projects (yes,
there’s a big difference between the two, I don’t know
at which end of the spectrum we will be at the end of
the year but our infrastructure will have to at least
be capable of scaling to the latter within two years)
with upwards of 2000 developers, drawing code from a
variety of sources, playing with it and spitting it
out regularly in nice packages.

Arch and Subversion seem to be the two leading
contenders for “next generation open source sccm”. I’d
be interested in your thoughts on the two of them, and
how they stack up. I’m looking to hire one person who
will lead that part of the effort. They’ll work alone
from home, and be responsible for two things. First,
extending the tool (arch or svn) in ways that help the
project. Such extensions will be released under an
open source licence, and hopefully embraced by the
tools maintainers and included in the mainline code
for the tool. And second, they will be responsible for
our large-scale implementation of SCCM, using that
tool, and building the management scripts and other
infrastructure to support such a large, and hopefully
highly automated, set of repositories.

Would you be interested in this position? What
attributes and experience do you think would make you
a great person to have on the team? What would your
salary expectation be, as a monthly figure, for a one
year contract full time?

I’m currently on your continent, well, just off it. On
Lizard Island, up North. Am headed today for Brisbane,
then on the 17th to Launceston via Melbourne. If you
happen to be on any of those stops, would you be
interested in meeting up to discuss it further?

If you’re curious you can find out a bit more about me
at www.markshuttleworth.com. This project is much
lower key than some of what you’ll find there. It’s a
very long shot indeed. But if at worst all that
happens is a bunch of open source work gets funded at
my expense I’ll feel it was money well spent.

Cheers,
Mark

=====

“Good judgement comes from experience, and often experience
comes from bad judgement” – Rita Mae Brown


,

Arjen LentzClassic McEleice and the NIST search for post-quantum crypto

I have always liked cryptography, and public-key cryptography in particularly. When Pretty Good Privacy (PGP) first came out in 1991, I not only started using it, also but looking at the documentation and the code to see how it worked. I created my own implementation in C using very small keys, just to better understand.

Cryptography has been running a race against both faster and cheaper computing power. And these days, with banking and most other aspects of our lives entirely relying on secure communications, it’s a very juicy target for bad actors.

About 5 years ago, the National (USA) Institute for Science and Technology (NIST) initiated a search for cryptographic algorithmic that should withstand a near-future world where quantum computers with a significant number of qubits are a reality. There have been a number of rounds, which mid 2020 saw round 3 and the finalists.

This submission caught my eye some time ago: Classic McEliece, and out of the four finalists it’s the only one that is not lattice-based [wikipedia link].

For Public Key Encryption and Key Exchange Mechanism, Prof Bill Buchanan thinks that the winner will be lattice-based, but I am not convinced.

Robert McEleice at his retirement in 2007

Tiny side-track, you may wonder where does the McEleice name come from? From mathematician Robert McEleice (1942-2019). McEleice developed his cryptosystem in 1978. So it’s not just named after him, he designed it. For various reasons that have nothing to do with the mathematical solidity of the ideas, it didn’t get used at the time. He’s done plenty cool other things, too. From his Caltech obituary:

He made fundamental contributions to the theory and design of channel codes for communication systems—including the interplanetary telecommunication systems that were used by the Voyager, Galileo, Mars Pathfinder, Cassini, and Mars Exploration Rover missions.

Back to lattices, there are both unknowns (aspects that have not been studied in exhaustive depth) and recent mathematical attacks, both of which create uncertainty – in the crypto sphere as well as for business and politics. Given how long it takes for crypto schemes to get widely adopted, the latter two are somewhat relevant, particularly since cyber security is a hot topic.

Lattices are definitely interesting, but given what we know so far, it is my feeling that systems based on lattices are more likely to be proven breakable than Classic McEleice, which come to this finalists’ table with 40+ years track record of in-depth analysis. Mind that all finalists are of course solid at this stage – but NIST’s thoughts on expected developments and breakthroughs is what is likely to decide the winner. NIST are not looking for shiny, they are looking for very very solid in all possible ways.

Prof Buchanan recently published implementations for the finalists, and did some benchmarks where we can directly compare them against each other.

We can see that Classic McEleice’s key generation is CPU intensive, but is that really a problem? The large size of its public key may be more of a factor (disadvantage), however the small ciphertext I think more than offsets that disadvantage.

As we’re nearing the end of the NIST process, in my opinion, fast encryption/decryption and small cyphertext, combined with the long track record of in-depth analysis, may still see Classic McEleice come out the winner.

The post Classic McEleice and the NIST search for post-quantum crypto first appeared on Lentz family blog.

,

Ian BrownKubenetes Basic Setup

GKE in Production - Part 1 This tutorial is part of a series I am creating on creating, running and managing Kubernetes on GCP the way I do in my day job. Note: There may be some things I have skimmed over, if so or you see a glaring hole in my configuration, please drop me a line via the contact page linked at the top of the site. What we will build In this first tutorial, we will be building a standard GKE cluster on Google Cloud Platform and deploying the hello world container to confirm everything is working.

,

Dave HallA Rube Goldberg Machine for Container Workflows

Learn how can you securely copy container images from GHCR to ECR.

,

Chris NeugebauerAdding a PurpleAir monitor to Home Assistant

Living in California, I’ve (sadly) grown accustomed to needing to keep track of our local air quality index (AQI) ratings, particularly as we live close to places where large wildfires happen every other year.

Last year, Josh and I bought a PurpleAir outdoor air quality meter, which has been great. We contribute our data to a collection of very local air quality meters, which is important, since the hilly nature of the North Bay means that the nearest government air quality ratings can be significantly different to what we experience here in Petaluma.

I recently went looking to pull my PurpleAir sensor data into my Home Assistant setup. Unfortunately, the PurpleAir API does not return the AQI metric for air quality, only the raw PM2.5/PM5/PM10 numbers. After some searching, I found a nice template sensor solution on the Home Assistant forums, which I’ve modernised by adding the AQI as a sub-sensor, and adding unique ID fields to each useful sensor, so that you can assign them to a location.

You’ll end up with sensors for raw PM2.5, the PM2.5 AQI value, the US EPA air quality category, air pressure, relative humidity and air pressure.

How to use this

First up, visit the PurpleAir Map, find the sensor you care about, click “get this widget�, and then “JSON�. That will give you the URL to set as the resource key in purpleair.yaml.

Adding the configuration

In HomeAssistant, add the following line to your configuration.yaml:

sensor: !include purpleair.yaml

and then add the following contents to purpleair.yaml


 - platform: rest
   name: 'PurpleAir'

   # Substitute in the URL of the sensor you care about.  To find the URL, go
   # to purpleair.com/map, find your sensor, click on it, click on "Get This
   # Widget" then click on "JSON".
   resource: https://www.purpleair.com/json?key={KEY_GOES_HERE}&show={SENSOR_ID}

   # Only query once a minute to avoid rate limits:
   scan_interval: 60

   # Set this sensor to be the AQI value.
   #
   # Code translated from JavaScript found at:
   # https://docs.google.com/document/d/15ijz94dXJ-YAZLi9iZ_RaBwrZ4KtYeCy08goGBwnbCU/edit#
   value_template: >
     {{ value_json["results"][0]["Label"] }}
   unit_of_measurement: ""
   # The value of the sensor can't be longer than 255 characters, but the
   # attributes can.  Store away all the data for use by the templates below.
   json_attributes:
     - results

 - platform: template
   sensors:
     purpleair_aqi:
       unique_id: 'purpleair_SENSORID_aqi_pm25'
       friendly_name: 'PurpleAir PM2.5 AQI'
       value_template: >
         {% macro calcAQI(Cp, Ih, Il, BPh, BPl) -%}
           {{ (((Ih - Il)/(BPh - BPl)) * (Cp - BPl) + Il)|round|float }}
         {%- endmacro %}
         {% if (states('sensor.purpleair_pm25')|float) > 1000 %}
           invalid
         {% elif (states('sensor.purpleair_pm25')|float) > 350.5 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 500.0, 401.0, 500.0, 350.5) }}
         {% elif (states('sensor.purpleair_pm25')|float) > 250.5 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 400.0, 301.0, 350.4, 250.5) }}
         {% elif (states('sensor.purpleair_pm25')|float) > 150.5 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 300.0, 201.0, 250.4, 150.5) }}
         {% elif (states('sensor.purpleair_pm25')|float) > 55.5 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 200.0, 151.0, 150.4, 55.5) }}
         {% elif (states('sensor.purpleair_pm25')|float) > 35.5 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 150.0, 101.0, 55.4, 35.5) }}
         {% elif (states('sensor.purpleair_pm25')|float) > 12.1 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 100.0, 51.0, 35.4, 12.1) }}
         {% elif (states('sensor.purpleair_pm25')|float) >= 0.0 %}
           {{ calcAQI((states('sensor.purpleair_pm25')|float), 50.0, 0.0, 12.0, 0.0) }}
         {% else %}
           invalid
         {% endif %}
       unit_of_measurement: "bit"
     purpleair_description:
       unique_id: 'purpleair_SENSORID_description'
       friendly_name: 'PurpleAir AQI Description'
       value_template: >
         {% if (states('sensor.purpleair_aqi')|float) >= 401.0 %}
           Hazardous
         {% elif (states('sensor.purpleair_aqi')|float) >= 301.0 %}
           Hazardous
         {% elif (states('sensor.purpleair_aqi')|float) >= 201.0 %}
           Very Unhealthy
         {% elif (states('sensor.purpleair_aqi')|float) >= 151.0 %}
           Unhealthy
         {% elif (states('sensor.purpleair_aqi')|float) >= 101.0 %}
           Unhealthy for Sensitive Groups
         {% elif (states('sensor.purpleair_aqi')|float) >= 51.0 %}
           Moderate
         {% elif (states('sensor.purpleair_aqi')|float) >= 0.0 %}
           Good
         {% else %}
           undefined
         {% endif %}
       entity_id: sensor.purpleair
     purpleair_pm25:
       unique_id: 'purpleair_SENSORID_pm25'
       friendly_name: 'PurpleAir PM 2.5'
       value_template: "{{ state_attr('sensor.purpleair','results')[0]['PM2_5Value'] }}"
       unit_of_measurement: "μg/m3"
       entity_id: sensor.purpleair
     purpleair_temp:
       unique_id: 'purpleair_SENSORID_temperature'
       friendly_name: 'PurpleAir Temperature'
       value_template: "{{ state_attr('sensor.purpleair','results')[0]['temp_f'] }}"
       unit_of_measurement: "°F"
       entity_id: sensor.purpleair
     purpleair_humidity:
       unique_id: 'purpleair_SENSORID_humidity'
       friendly_name: 'PurpleAir Humidity'
       value_template: "{{ state_attr('sensor.purpleair','results')[0]['humidity'] }}"
       unit_of_measurement: "%"
       entity_id: sensor.purpleair
     purpleair_pressure:
       unique_id: 'purpleair_SENSORID_pressure'
       friendly_name: 'PurpleAir Pressure'
       value_template: "{{ state_attr('sensor.purpleair','results')[0]['pressure'] }}"
       unit_of_measurement: "hPa"
       entity_id: sensor.purpleair

Quirks

I had difficulty getting the AQI to display as a numeric graph when I didn’t set a unit. I went with bit, and that worked just fine. 🤷�♂�

,

Stewart SmithAn Unearthly Child

So, this idea has been brewing for a while now… try and watch all of Doctor Who. All of it. All 38 seasons. Today(ish), we started. First up, from 1963 (first aired not quite when intended due to the Kennedy assassination): An Unearthly Child. The first episode of the first serial.

A lot of iconic things are there from the start: the music, the Police Box, embarrassing moments of not quite remembering what time one is in, and normal humans accidentally finding their way into the TARDIS.

I first saw this way back when a child, where they were repeated on ABC TV in Australia for some anniversary of Doctor Who (I forget which one). Well, I saw all but the first episode as the train home was delayed and stopped outside Caulfield for no reason for ages. Some things never change.

Of course, being a show from the early 1960s, there’s some rougher spots. We’re not about to have the picture of diversity, and there’s going to be casual racism and sexism. What will be interesting is noticing these things today, and contrasting with my memory of them at the time (at least for episodes I’ve seen before), and what I know of the attitudes of the time.

“This year-ometer is not calculating properly” is a very 2020 line though (technically from the second episode).

,

Jan SchmidtRift CV1 – Getting close now…

It’s been a while since my last post about tracking support for the Oculus Rift in February. There’s been big improvements since then – working really well a lot of the time. It’s gone from “If I don’t make any sudden moves, I can finish an easy Beat Saber level” to “You can’t hide from me!” quality.

Equally, there are still enough glitches and corner cases that I think I’ll still be at this a while.

Here’s a video from 3 weeks ago of (not me) playing Beat Saber on Expert+ setting showing just how good things can be now:

Beat Saber – Skunkynator playing Expert+, Mar 16 2021

Strap in. Here’s what I’ve worked on in the last 6 weeks:

Pose Matching improvements

Most of the biggest improvements have come from improving the computer vision algorithm that’s matching the observed LEDs (blobs) in the camera frames to the 3D models of the devices.

I split the brute-force search algorithm into 2 phases. It now does a first pass looking for ‘obvious’ matches. In that pass, it does a shallow graph search of blobs and their nearest few neighbours against LEDs and their nearest neighbours, looking for a match using a “Strong” match metric. A match is considered strong if expected LEDs match observed blobs to within 1.5 pixels.

Coupled with checks on the expected orientation (matching the Gravity vector detected by the IMU) and the pose prior (expected position and orientation are within predicted error bounds) this short-circuit on the search is hit a lot of the time, and often completes within 1 frame duration.

In the remaining tricky cases, where a deeper graph search is required in order to recover the pose, the initial search reduces the number of LEDs and blobs under consideration, speeding up the remaining search.

I also added an LED size model to the mix – for a candidate pose, it tries to work out how large (in pixels) each LED should appear, and use that as a bound on matching blobs to LEDs. This helps reduce mismatches as devices move further from the camera.

LED labelling

When a brute-force search for pose recovery completes, the system now knows the identity of various blobs in the camera image. One way it avoids a search next time is to transfer the labels into future camera observations using optical-flow tracking on the visible blobs.

The problem is that even sped-up the search can still take a few frame-durations to complete. Previously LED labels would be transferred from frame to frame as they arrived, but there’s now a unique ID associated with each blob that allows the labels to be transferred even several frames later once their identity is known.

IMU Gyro scale

One of the problems with reverse engineering is the guesswork around exactly what different values mean. I was looking into why the controller movement felt “swimmy” under fast motions, and one thing I found was that the interpretation of the gyroscope readings from the IMU was incorrect.

The touch controllers report IMU angular velocity readings directly as a 16-bit signed integer. Previously the code would take the reading and divide by 1024 and use the value as radians/second.

From teardowns of the controller, I know the IMU is an Invensense MPU-6500. From the datasheet, the reported value is actually in degrees per second and appears to be configured for the +/- 2000 °/s range. That yields a calculation of Gyro-rad/s = Gyro-°/s * (2000 / 32768) * (?/180) – or a divisor of 938.734.

The 1024 divisor was under-estimating rotation speed by about 10% – close enough to work until you start moving quickly.

Limited interpolation

If we don’t find a device in the camera views, the fusion filter predicts motion using the IMU readings – but that quickly becomes inaccurate. In the worst case, the controllers fly off into the distance. To avoid that, I added a limit of 500ms for ‘coasting’. If we haven’t recovered the device pose by then, the position is frozen in place and only rotation is updated until the cameras find it again.

Exponential filtering

I implemented a 1-Euro exponential smoothing filter on the output poses for each device. This is an idea from the Project Esky driver for Project North Star/Deck-X AR headsets, and almost completely eliminates jitter in the headset view and hand controllers shown to the user. The tradeoff is against introducing lag when the user moves quickly – but there are some tunables in the exponential filter to play with for minimising that. For now I’ve picked some values that seem to work reasonably.

Non-blocking radio

Communications with the touch controllers happens through USB radio command packets sent to the headset. The main use of radio commands in OpenHMD is to read the JSON configuration block for each controller that is programmed in at the factory. The configuration block provides the 3D model of LED positions as well as initial IMU bias values.

Unfortunately, reading the configuration block takes a couple of seconds on startup, and blocks everything while it’s happening. Oculus saw that problem and added a checksum in the controller firmware. You can read the checksum first and if it hasn’t changed use a local cache of the configuration block. Eventually, I’ll implement that caching mechanism for OpenHMD but in the meantime it still reads the configuration blocks on each startup.

As an interim improvement I rewrote the radio communication logic to use a state machine that is checked in the update loop – allowing radio communications to be interleaved without blocking the regularly processing of events. It still interferes a bit, but no longer causes a full multi-second stall as each hand controller turns on.

Haptic feedback

The hand controllers have haptic feedback ‘rumble’ motors that really add to the immersiveness of VR by letting you sense collisions with objects. Until now, OpenHMD hasn’t had any support for applications to trigger haptic events. I spent a bit of time looking at USB packet traces with Philipp Zabel and we figured out the radio commands to turn the rumble motors on and off.

In the Rift CV1, the haptic motors have a mode where you schedule feedback events into a ringbuffer – effectively they operate like a low frequency audio device. However, that mode was removed for the Rift S (and presumably in the Quest devices) – and deprecated for the CV1.

With that in mind, I aimed for implementing the unbuffered mode, with explicit ‘motor on + frequency + amplitude’ and ‘motor off’ commands sent as needed. Thanks to already having rewritten the radio communications to use a state machine, adding haptic commands was fairly easy.

The big question mark is around what API OpenHMD should provide for haptic feedback. I’ve implemented something simple for now, to get some discussion going. It works really well and adds hugely to the experience. That code is in the https://github.com/thaytan/OpenHMD/tree/rift-haptics branch, with a SteamVR-OpenHMD branch that uses it in https://github.com/thaytan/SteamVR-OpenHMD/tree/controller-haptics-wip

Problem areas

Unexpected tracking losses

I’d say the biggest problem right now is unexpected tracking loss and incorrect pose extractions when I’m not expecting them. Especially my right controller will suddenly glitch and start jumping around. Looking at a video of the debug feed, it’s not obvious why that’s happening:

To fix cases like those, I plan to add code to log the raw video feed and the IMU information together so that I can replay the video analysis frame-by-frame and investigate glitches systematically. Those recordings will also work as a regression suite to test future changes.

Sensor fusion efficiency

The Kalman filter I have implemented works really nicely – it does the latency compensation, predicts motion and extracts sensor biases all in one place… but it has a big downside of being quite expensive in CPU. The Unscented Kalman Filter CPU cost grows at O(n^3) with the size of the state, and the state in this case is 43 dimensional – 22 base dimensions, and 7 per latency-compensation slot. Running 1000 updates per second for the HMD and 500 for each of the hand controllers adds up quickly.

At some point, I want to find a better / cheaper approach to the problem that still provides low-latency motion predictions for the user while still providing the same benefits around latency compensation and bias extraction.

Lens Distortion

To generate a convincing illusion of objects at a distance in a headset that’s only a few centimetres deep, VR headsets use some interesting optics. The LCD/OLED panels displaying the output get distorted heavily before they hit the users eyes. What the software generates needs to compensate by applying the right inverse distortion to the output video.

Everyone that tests the CV1 notices that the distortion is not quite correct. As you look around, the world warps and shifts annoyingly. Sooner or later that needs fixing. That’s done by taking photos of calibration patterns through the headset lenses and generating a distortion model.

Camera / USB failures

The camera feeds are captured using a custom user-space UVC driver implementation that knows how to set up the special synchronisation settings of the CV1 and DK2 cameras, and then repeatedly schedules isochronous USB packet transfers to receive the video.

Occasionally, some people experience failure to re-schedule those transfers. The kernel rejects them with an out-of-memory error failing to set aside DMA memory (even though it may have been running fine for quite some time). It’s not clear why that happens – but the end result at the moment is that the USB traffic for that camera dies completely and there’ll be no more tracking from that camera until the application is restarted.

Often once it starts happening, it will keep happening until the PC is rebooted and the kernel memory state is reset.

Occluded cases

Tracking generally works well when the cameras get a clear shot of each device, but there are cases like sighting down the barrel of a gun where we expect that the user will line up the controllers in front of one another, and in front of the headset. In that case, even though we probably have a good idea where each device is, it can be hard to figure out which LEDs belong to which device.

If we already have a good tracking lock on the devices, I think it should be possible to keep tracking even down to 1 or 2 LEDs being visible – but the pose assessment code will have to be aware that’s what is happening.

Upstreaming

April 14th marks 2 years since I first branched off OpenHMD master to start working on CV1 tracking. How hard can it be, I thought? I’ll knock this over in a few months.

Since then I’ve accumulated over 300 commits on top of OpenHMD master that eventually all need upstreaming in some way.

One thing people have expressed as a prerequisite for upstreaming is to try and remove the OpenCV dependency. The tracking relies on OpenCV to do camera distortion calculations, and for their PnP implementation. It should be possible to reimplement both of those directly in OpenHMD with a bit of work – possibly using the fast LambdaTwist P3P algorithm that Philipp Zabel wrote, that I’m already using for pose extraction in the brute-force search.

Others

I’ve picked the top issues to highlight here. https://github.com/thaytan/OpenHMD/issues has a list of all the other things that are still on the radar for fixing eventually.

Other Headsets

At some point soon, I plan to put a pin in the CV1 tracking and look at adapting it to more recent inside-out headsets like the Rift S and WMR headsets. I implemented 3DOF support for the Rift S last year, but getting to full positional tracking for that and other inside-out headsets means implementing a SLAM/VIO tracking algorithm to track the headset position.

Once the headset is tracking, the code I’m developing here for CV1 to find and track controllers will hopefully transfer across – the difference with inside-out tracking is that the cameras move around with the headset. Finding the controllers in the actual video feed should work much the same.

Sponsorship

This development happens mostly in my spare time and partly as open source contribution time at work at Centricular. I am accepting funding through Github Sponsorships to help me spend more time on it – I’d really like to keep helping Linux have top-notch support for VR/AR applications. Big thanks to the people that have helped get this far.

,

Stewart Smithlibeatmydata v129

Every so often, I release a new libeatmydata. This has not happened for a long time. This is just some bug fixes, most of which have been in the Debian package for some time, I’ve just been lazy and not sat down and merged them.

git clone https://github.com/stewartsmith/libeatmydata.git

Download the source tarball from here: libeatmydata-129.tar.gz and GPG signature: libeatmydata-129.tar.gz.asc from my GPG key.

Or, feel free to grab some Fedora RPMs:

Releases published also in the usual places:

,

BlueHackersWorld bipolar day 2021

Today, 30 March, is World Bipolar Day.

Vincent van Gogh - Worn Out

Why that particular date? It’s Vincent van Gogh’s birthday (1853), and there is a fairly strong argument that the Dutch painter suffered from bipolar (among other things).

The image on the side is Vincent’s drawing “Worn Out” (from 1882), and it seems to capture the feeling rather well – whether (hypo)manic, depressed, or mixed. It’s exhausting.

Bipolar is complicated, often undiagnosed or misdiagnosed, and when only treated with anti-depressants, it can trigger the (hypo)mania – essentially dragging that person into that state near-permanently.

Have you heard of Bipolar II?

Hypo-mania is the “lesser” form of mania that distinguishes Bipolar I (the classic “manic depressive” syndrome) from Bipolar II. It’s “lesser” only in the sense that rather than someone going so hyper they may think they can fly (Bipolar I is often identified when someone in manic state gets admitted to hospital – good catch!) while with Bipolar II the hypo-mania may actually exhibit as anger. Anger in general, against nothing in particular but potentially everyone and everything around them. Or, if it’s a mixed episode, anger combined with strong negative thoughts. Either way, it does not look like classic mania. It is, however, exhausting and can be very debilitating.

Bipolar II people often present to a doctor while in depressed state, and GPs (not being psychiatrists) may not do a full diagnosis. Note that D.A.S. and similar test sheets are screening tools, they are not diagnostic. A proper diagnosis is more complex than filling in a form some questions (who would have thought!)

Call to action

If you have a diagnosis of depression, only from a GP, and are on medication for this, I would strongly recommend you also get a referral to a psychiatrist to confirm that diagnosis.

Our friends at the awesome Black Dog Institute have excellent information on bipolar, as well as a quick self-test – if that shows some likelihood of bipolar, go get that referral and follow up ASAP.

I will be writing more about the topic in the coming time.

The post World bipolar day 2021 first appeared on BlueHackers.org.

,

Stewart SmithThe Apple Power Macintosh 7200/120 PC Compatible (Part 1)

So, I learned something recently: if you pick up your iPhone with eBay open on an auction bid screen in just the right way, you may accidentally click the bid button and end up buying an old computer. Totally not the worst thing ever, and certainly a creative way to make a decision.

So, not too long later, a box arrives!

In the 1990s, Apple created some pretty “interesting” computers and product line. One thing you could get is a DOS Compatibility (or PC Compatibility) card. This was a card that went into one of the expansion slots on a Mac and had something really curious on it: most of the guts of a PC.

Others have written on these cards too: https://www.engadget.com/2009-12-10-before-there-was-boot-camp-there-were-dos-compatibility-cards.html and http://www.edibleapple.com/2009/12/09/blast-from-the-past-a-look-back-at-apples-dos-compatibility-cards/. There’s also the Service Manual https://tim.id.au/laptops/apple/misc/pc_compatibility_card.pdf with some interesting details.

The machine I’d bought was an Apple Power Macintosh 7200/120 with the PC Compatible card added afterwards (so it doesn’t have the PC Compatible label on the front like some models ended up getting).

The Apple Power Macintosh 7200/120

Wikipedia has a good article on the line, noting that it was first released in August 1995, and fitting for the era, was sold as about 14 million other model numbers (okay not quite that bad, it was only a total of four model numbers for essentially the same machine). This specific model, the 7200/120 was introduced on April 22nd, 1996, and the original web page describing it from Apple is on the wayback machine.

For older Macs, Low End Mac is a good resource, and there’s a page on the 7200, and amazingly Apple still has the tech specs on their web site!

The 7200 series replaced the 7100, which was one of the original PowerPC based Macs. The big changes are using the industry standard PCI bus for its three expansion slots rather than NuBus. Rather surprisingly, NuBus was not Apple specific, but you could not call it widely adopted by successful manufacturers. Apple first used NuBus in the 1987 Macintosh II.

The PCI bus was standardized in 1992, and it’s almost certain that a successor to it is in the computer you’re using to read this. It really quite caught on as an industry standard.

The processor of the machine is a PowerPC 601. The PowerPC was an effort of IBM, Apple, and Motorola (the AIM Alliance) to create a class of processors for personal computers based on IBM’s POWER Architecture. The PowerPC 601 was the first of these processors, initially used by Apple in its Power Macintosh range. The machine I have has one running at a whopping 120Mhz. There continued to be PowerPC chips for a number of years, and IBM continued making POWER processors even after that. However, you are almost certainly not using a PowerPC derived processor in the computer you’re using to read this.

The PC Compatibility card has on it a full on legit Pentium 100 processor, and hardware for doing VGA graphics, a Sound Blaster 16 and the other things you’d usually expect of a PC from 1996. Since it’s on a PCI card though, it’s a bit different than a PC of the era. It doesn’t have any expansion slots of its own, and in fact uses up one of the three PCI slots in the Mac. It also doesn’t have its own floppy drive, or hard drive. There’s software on the Mac that will let the PC card use the Mac’s floppy drive, and part of the Mac’s hard drive for the PC!

The Pentium 100 was the first mass produced superscalar processor. You are quite likely to be using a computer with a processor related to the Pentium to read this, unless you’re using a phone or tablet, or one of the very latest Macs; in which case you’re using an ARM based processor. You likely have more ARM processors in your life than you have socks.

Basically, this computer is a bit of a hodge-podge of historical technology, some of which ended up being successful, and other things less so.

Let’s have a look inside!

So, one of the PCI slots has a Vertex Twin Turbo 128M8A video card in it. There is not much about this card on the internet. There’s a photo of one on Wikimedia Commons though. I’ll have to investigate more.

Does it work though? Yes! Here it is on my desk:

The powered on Power Mac 7200/120

Even with Microsoft Internet Explorer 4.0 that came with MacOS 8.6, you can find some places on the internet you can fetch files from, at a not too bad speed even!

More fun times with this machine to come!

,

Dave HallParameter Store vs Secrets Manager

Which AWS managed service is best for storing and managing your secrets?

,

Dave HallA Lost Parcel Results in a New Website

When Australia Post lost a parcel, we found a lot of problems with one of their websites.

,

Jan SchmidtRift CV1 – Testing SteamVR

Update:

This post documented an older method of building SteamVR-OpenHMD. I moved them to a page here. That version will be kept up to date for any future changes, so go there.


I’ve had a few people ask how to test my OpenHMD development branch of Rift CV1 positional tracking in SteamVR. Here’s what I do:

  • Make sure Steam + SteamVR are already installed.
  • Clone the SteamVR-OpenHMD repository:
git clone --recursive https://github.com/ChristophHaag/SteamVR-OpenHMD.git
  • Switch the internal copy of OpenHMD to the right branch:
cd subprojects/openhmd
git remote add thaytan-github https://github.com/thaytan/OpenHMD.git
git fetch thaytan-github
git checkout -b rift-kalman-filter thaytan-github/rift-kalman-filter
cd ../../
  • Use meson to build and register the SteamVR-OpenHMD binaries. You may need to install meson first (see below):
meson -Dbuildtype=release build
ninja -C build
./install_files_to_build.sh
./register.sh
  • It is important to configure in release mode, as the kalman filtering code is generally too slow for real-time in debug mode (it has to run 2000 times per second)
  • Make sure your USB devices are accessible to your user account by configuring udev. See the OpenHMD guide here: https://github.com/OpenHMD/OpenHMD/wiki/Udev-rules-list
  • Please note – only Rift sensors on USB 3.0 ports will work right now. Supporting cameras on USB 2.0 requires someone implementing JPEG format streaming and decoding.
  • It can be helpful to test OpenHMD is working by running the simple example. Check that it’s finding camera sensors at startup, and that the position seems to change when you move the headset:
./build/subprojects/openhmd/openhmd_simple_example
  • Calibrate your expectations for how well tracking is working right now! Hint: It’s very experimental 🙂
  • Start SteamVR. Hopefully it should detect your headset and the light(s) on your Rift Sensor(s) should power on.

Meson

I prefer the Meson build system here. There’s also a cmake build for SteamVR-OpenHMD you can use instead, but I haven’t tested it in a while and it sometimes breaks as I work on my development branch.

If you need to install meson, there are instructions here – https://mesonbuild.com/Getting-meson.html summarising the various methods.

I use a copy in my home directory, but you need to make sure ~/.local/bin is in your PATH

pip3 install --user meson

,

Jan SchmidtRift CV1 – Pose rejection

I spent some time this weekend implementing a couple of my ideas for improving the way the tracking code in OpenHMD filters and rejects (or accepts) possible poses when trying to match visible LEDs to the 3D models for each device.

In general, the tracking proceeds in several steps (in parallel for each of the 3 devices being tracked):

  1. Do a brute-force search to match LEDs to 3D models, then (if matched)
    1. Assign labels to each LED blob in the video frame saying what LED they are.
    2. Send an update to the fusion filter about the position / orientation of the device
  2. Then, as each video frame arrives:
    1. Use motion flow between video frames to track the movement of each visible LED
    2. Use the IMU + vision fusion filter to predict the position/orientation (pose) of each device, and calculate which LEDs are expected to be visible and where.
  3. Try and match up and refine the poses using the predicted pose prior and labelled LEDs. In the best case, the LEDs are exactly where the fusion predicts they’ll be. More often, the orientation is mostly correct, but the position has drifted and needs correcting. In the worst case, we send the frame back to step 1 and do a brute-force search to reacquire an object.

The goal is to always assign the correct LEDs to the correct device (so you don’t end up with the right controller in your left hand), and to avoid going back to the expensive brute-force search to re-acquire devices as much as possible

What I’ve been working on this week is steps 1 and 3 – initial acquisition of correct poses, and fast validation / refinement of the pose in each video frame, and I’ve implemented two new strategies for that.

Gravity Vector matching

The first new strategy is to reject candidate poses that don’t closely match the known direction of gravity for each device. I had a previous implementation of that idea which turned out to be wrong, so I’ve re-worked it and it helps a lot with device acquisition.

The IMU accelerometer and gyro can usually tell us which way up the device is (roll and pitch) but not which way they are facing (yaw). The measure for ‘known gravity’ comes from the fusion Kalman filter covariance matrix – how certain the filter is about the orientation of the device. If that variance is small this new strategy is used to reject possible poses that don’t have the same idea of gravity (while permitting rotations around the Y axis), with the filter variance as a tolerance.

Partial tracking matches

The 2nd strategy is based around tracking with fewer LED correspondences once a tracking lock is acquired. Initial acquisition of the device pose relies on some heuristics for how many LEDs must match the 3D model. The general heuristic threshold I settled on for now is that 2/3rds of the expected LEDs must be visible to acquire a cold lock.

With the new strategy, if the pose prior has a good idea where the device is and which way it’s facing, it allows matching on far fewer LED correspondences. The idea is to keep tracking a device even down to just a couple of LEDs, and hope that more become visible soon.

While this definitely seems to help, I think the approach can use more work.

Status

With these two new approaches, tracking is improved but still quite erratic. Tracking of the headset itself is quite good now and for me rarely loses tracking lock. The controllers are better, but have a tendency to “fly off my hands” unexpectedly, especially after fast motions.

I have ideas for more tracking heuristics to implement, and I expect a continuous cycle of refinement on the existing strategies and new ones for some time to come.

For now, here’s a video of me playing Beat Saber using tonight’s code. The video shows the debug stream that OpenHMD can generate via Pipewire, showing the camera feed plus overlays of device predictions, LED device assignments and tracked device positions. Red is the headset, Green is the right controller, Blue is the left controller.

Initial tracking is completely wrong – I see some things to fix there. When the controllers go offline due to inactivity, the code keeps trying to match LEDs to them for example, and then there are some things wrong with how it’s relabelling LEDs when they get incorrect assignments.

After that, there are periods of good tracking with random tracking losses on the controllers – those show the problem cases to concentrate on.

,

Colin CharlesLife with Rona 2.0 – Days 4, 5, 6, 7, 8 and 9

These lack of updates are also likely because I’ve been quite caught up with stuff.

Monday I had a steak from Bay Leaf Steakhouse for dinner. It was kind of weird eating it from packs, but then I’m reminded you could do this in economy class. Tuesday I wanted to attempt to go vegetarian and by the time I was done with a workout, the only place was a chap fan shop (Leong Heng) where I had a mixture of Chinese and Indian chap fan. The Indian stall is run by an ex-Hyatt staff member who immediately recognised me! Wednesday, Alice came to visit, so we got to Hanks, got some alcohol, and managed a smorgasbord of food from Pickers/Sate Zul/Lila Wadi. Night ended very late, and on Thursday, visited Hai Tian for their famous salted egg squid and prawns in a coconut shell. Friday was back to being normal, so I grabbed a pizza from Mint Pizza (this time I tried their Aussie variant). Saturday, today, I hit up Rasa Sayang for some matcha latte, but grabbed food from Classic Pilot Cafe, which Faeeza owns! It was the famous salted egg chicken, double portion, half rice.

As for workouts, I did sign up for Mantas but found it pretty hard to do, timezone wise. I did spend a lot of time jogging on the beach (this has been almost a daily affair). Monday I also did 2 MD workouts, Tuesday 1 MD workout, Wednesday half a MD workout, Thursday I did a Ping workout at Pwrhouse (so good!), Friday 1 MD workout, and Saturday an Audrey workout at Pwrhouse and 1 MD workout.

Wednesday I also found out that Rasmus passed away. Frankly, there are no words.

Thursday, my Raspberry Pi 400 arrived. I set it up in under ten minutes, connecting it to the TV here. It “just works”. I made a video, which I should probably figure out how to upload to YouTube after I stitch it together. I have to work on using it a lot more.

COVID-19 cases are through the roof in Malaysia. This weekend we’ve seen two days of case breaking records, with today being 5,728 (yesterday was something close). Nutty. Singapore suspended the reciprocal green lane (RGL) agreement with Malaysia for the next 3 months.

I’ve managed to finish Bridgerton. I like the score. Finding something on Netflix is proving to be more difficult, regardless of having a VPN. Honestly, this is why Cable TV wins… linear programming that you’re just fed.

Stock market wise, I’ve been following the GameStop short squeeze, and even funnier is the Top Glove one, that they’re trying to repeat in Malaysia. Bitcoin seems to be doing “reasonably well” and I have to say, I think people are starting to realise decentralised services have a future. How do we get there?

What an interesting week, I look forward to more productive time. I’m still writing in my Hobonichi Techo, so at least that’s where most personal stuff ends up, I guess?

The post Life with Rona 2.0 – Days 4, 5, 6, 7, 8 and 9 first appeared on Colin Charles Agenda.

,

Jan SchmidtHitting a milestone – Beat Saber!

I hit an important OpenHMD milestone tonight – I completed a Beat Saber level using my Oculus Rift CV1!

I’ve been continuing to work on integrating Kalman filtering into OpenHMD, and on improving the computer vision that matches and tracks device LEDs. While I suspect noone will be completing Expert levels just yet, it’s working well enough that I was able to play through a complete level of Beat Saber. For a long time this has been my mental benchmark for tracking performance, and I’m really happy 🙂

Check it out:

I should admit at this point that completing this level took me multiple attempts. The tracking still has quite a tendency to lose track of controllers, or to get them confused and swap hands suddenly.

I have a list of more things to work on. See you at the next update!

,

Colin CharlesLife with Rona 2.0 – Day 3

What an unplanned day. I woke up in time to do an MD workout, despite feeling a little sore. So maybe I was about 10 minutes late and I missed the first set, but his workouts are so long, and I think there were seven sets anyway. Had a good brunch shortly thereafter.

Did a bit of reading, and then I decided to do a beach boardwalk walk… turns out they were policing the place, and you can’t hit the boardwalk. But the beach is fair game? So I went back to the hotel, dropped off my slippers, and went for a beach jog. Pretty nutty.

Came back to read a little more and figured I might as well do another MD workout. Then I headed out for dinner, trying out a new place — Mint Pizza. Opened 20.12.2020, and they’re empty, and their pizza is actually pretty good. Lamb and BBQ chicken, they did half-and-half.

Twitter was discussing Raspberry Pi’s, and all I could see is a lot of misinformation, which is truly shocking. The irony is that open source has been running the Internet for so long, and progressive web apps have come such a long way…

Back in the day when I did OpenOffice.org or Linux training even, we always did say you should learn concepts and not tools. From the time we ran Linux installfests in the late-90s in Sunway Pyramid (back then, yes, Linux was hard, and you had winmodems), but I had forgotten that I even did stuff for school teachers and NGOs back in 2002… I won’t forget PC Gemilang either…

Anyway, I placed an order again for another Raspberry Pi 400. I am certain that most people talk so much crap, without realising that Malaysia isn’t a developed nation and most people can’t afford a Mac let alone a PC. Laptops aren’t cheap. And there are so many other issues…. Saying Windows is still required in 2021 is the nuttiest thing I’ve heard in a long time. Easy to tweet, much harder to think about TCO, and realise where in the journey Malaysia is.

Maybe the best thing was that Malaysian Twitter learned about technology. I doubt many realised the difference between a Pi board vs the 400, but hey, the fact that they talked about tech is still a win (misinformed, but a win).

The post Life with Rona 2.0 – Day 3 first appeared on Colin Charles Agenda.

,

Colin CharlesLife with Rona 2.0 – Days 1 & 2

Today is the first day that in the state of Pahang, we have to encounter what many Malaysians are referring to as the Movement Control Order 2.0 (MCO 2.0). I think everyone finally agrees with the terminology that this is a lockdown now, because I remember back in the day when I was calling it that, I’d definitely offend a handful of journalists.

This is one interesting change for me compared to when I last wrote Life with RonaDay 56 of being indoors and not even leaving my household, in Kuala Lumpur. I am now not in the state, I am living in a hotel, and I am obviously moving around a little more since we have access to the beach.

KL/Selangor and several other states have already been under the MCO 2.0 since January 13 2021, and while it was supposed to end on January 26, it seems like they’ve extended and harmonised the dates for Peninsular Malaysia to end on February 4 2021. I guess everyone got the “good news” yesterday. The Prime Minister announced some kind of aid last week, but it is still mostly a joke.

Today was the 2nd day I woke up at around 2.30pm because I went to bed at around 8am. First day I had a 23.5 hour uptime, and the today was less brutal, but working from 1-8am with the PST timezone is pretty brutal. Consequently, I barely got too much done, and had one meal, vegetarian, two packs that included rice. I did get to walk by the beach (between Teluk Cempedak and Teluk Cempedak 2), did quite a bit of exercise there and I think even the monkeys are getting hungry… lots of stray cats and monkeys. Starbucks closes at 7pm, and I rocked up at 7.10pm (this was just like yesterday, when I arrived at 9.55pm and was told they wouldn’t grant me a coffee!).

While writing this entry, I did manage to get into a long video call with some friends and I guess it was good catching up with people in various states. It also is what prevented me from publishing this entry!

Day 2

I did wake up reasonable early today because I had pre-ordered room service to arrive at 9am. There is a fixed menu at the hotel for various cuisines (RM48/pax, thankfully gratis for me) and I told them I prefer not having to waste, so just give me what I want which is off menu items anyway. Roti telur double telur (yes, I know it is a roti jantan) with some banjir dhal and sambal and a bit of fruit on the side with two teh tariks. They delivered as requested. I did forget to ask for a jar of honey but that is OK, there is always tomorrow.

I spent most of the day vacillating, and wouldn’t consider it productive by any measure. Just chit chats and napping. It did rain today after a long time, so the day seemed fairly dreary.

When I finally did awaken from my nap, I went for a run on the beach. I did it barefoot. I have no idea if this is how it is supposed to be done, or if you are to run nearer the water or further up above, but I did move around between the two quite often. The beach is still pretty dead, but it is expected since no one is allowed to go unless you’re a hotel guest.

The hotel has closed 3/4 of their villages (blocks) and moved everyone to the village I’m staying in (for long stay guests…). I’m thankful I have a pretty large suite, it is a little over 980sqft, and the ample space, while smaller than my home, is still welcome.

Post beach run, I did a workout with MD via Instagram. It was strength/HIIT based, and I burnt a tonne, because he gave us one of his signature 1.5h classes. It was longer than the 80 minute class he normally charges RM50 for (I still think this is undervaluing his service, but he really does care and does it for the love of seeing his students grow!).

Post-workout I decided to head downtown to find some dinner. Everything at the Teluk Cemepdak block of shops was closed, so they’re not even bothered with doing takeaway. Sg. Lembing steakhouse seemed to have cars parked, Vanggey was empty (Crocodile Rock was open, can’t say if there was a crowd, because the shared parking lot was empty), there was a modest queue at Sate Zul, and further down, Lena was closed, Pickers was open for takeaway but looked pretty closed, Tjantek was open surprisingly, and then I thought I’d give Nusantara a try again, this time for food, but their chef had just gone home at about 8pm. Oops. So I drove to LAN burger, initially ordering just one chicken double special; however they looked like they could use the business so I added on a beef double special. They now accept Boost payments so have joined the e-wallet era. One less place to use cash, which is also why I really like Kuantan. On the drive back, Classic Pilot Cafe was also open and I guess I’ll be heading there too during this lockdown.

Came back to the room to finish both burgers in probably under 15 minutes. While watching the first episode of Bridgerton on Netflix. I’m not sure what really captivates, but I will continue on (I still haven’t finished the first episode). I need to figure out how to use the 2 TVs that I have in this room — HDMI cable? Apple TV? Not normally using a TV, all this is clearly more complex than I care to admit.

I soaked longer than expected, ended up a prune, but I’m sure it will give me good rest!

One thought to leave with:

“Learn to enjoy every minute of your life. Be happy now. Don’t wait for something outside of yourself to make you happy in the future.” — Earl Nightingale

The post Life with Rona 2.0 – Days 1 & 2 first appeared on Colin Charles Agenda.

,

Sam WatkinsDeveloping CZ, a dialect of C that looks like Python

In my experience, the C programming language is still hard to beat, even 50 years after it was first developed (and I feel the same way about UNIX). When it comes to general-purpose utility, low-level systems programming, performance, and portability (even to tiny embedded systems), I would choose C over most modern or fashionable alternatives. In some cases, it is almost the only choice.

Many developers believe that it is difficult to write secure and reliable software in C, due to its free pointers, the lack of enforced memory integrity, and the lack of automatic memory management; however in my opinion it is possible to overcome these risks with discipline and a more secure system of libraries constructed on top of C and libc. Daniel J. Bernstein and Wietse Venema are two developers who have been able to write highly secure, stable, reliable software in C.

My other favourite language is Python. Although Python has numerous desirable features, my favourite is the light-weight syntax: in Python, block structure is indicated by indentation, and braces and semicolons are not required. Apart from the pleasure and relief of reading and writing such light and clear code, which almost appears to be executable pseudo-code, there are many other benefits. In C or JavaScript, if you omit a trailing brace somewhere in the code, or insert an extra brace somewhere, the compiler may tell you that there is a syntax error at the end of the file. These errors can be annoying to track down, and cannot occur in Python. Python not only looks better, the clear syntax helps to avoid errors.

The obvious disadvantage of Python, and other dynamic interpreted languages, is that most programs run extremely slower than C programs. This limits the scope and generality of Python. No AAA or performance-oriented video game engines are programmed in Python. The language is not suitable for low-level systems programming, such as operating system development, device drivers, filesystems, performance-critical networking servers, or real-time systems.

C is a great all-purpose language, but the code is uglier than Python code. Once upon a time, when I was experimenting with the Plan 9 operating system (which is built on C, but lacks Python), I missed Python’s syntax, so I decided to do something about it and write a little preprocessor for C. This converts from a “Pythonesque” indented syntax to regular C with the braces and semicolons. Having forked a little dialect of my own, I continued from there adding other modules and features (which might have been a mistake, but it has been fun and rewarding).

At first I called this translator Brace, because it added in the braces for me. I now call the language CZ. It sounds like “C-easy”. Ease-of-use for developers (DX) is the primary goal. CZ has all of the features of C, and translates cleanly into C, which is then compiled to machine code as normal (using any C compiler; I didn’t write one); and so CZ has the same features and performance as C, but enjoys a more pleasing syntax.

CZ is now self-hosted, in that the translator is written in the language CZ. I confess that originally I wrote most of it in Perl; I’m proficient at Perl, but I consider it to be a fairly ugly language, and overly complicated.

I intend for CZ’s new syntax to be “optional”, ideally a developer will be able to choose to use the normal C syntax when editing CZ, if they prefer it. For this, I need a tool to convert C back to CZ, which I have not fully implemented yet. I am aware that, in addition to traditionalists, some vision-impaired developers prefer to use braces and semicolons, as screen readers might not clearly indicate indentation. A C to CZ translator would of course also be valuable when porting an existing C program to CZ.

CZ has a number of useful features that are not found in standard C, but I did not go so far as C++, which language has been described as “an octopus made by nailing extra legs onto a dog”. I do not consider C to be a dog, at least not in a negative sense; but I think that C++ is not an improvement over plain C. I am creating CZ because I think that it is possible to improve on C, without losing any of its advantages or making it too complex.

One of the most interesting features I added is a simple syntax for fast, light coroutines. I based this on Simon Tatham’s approach to Coroutines in C, which may seem hacky at first glance, but is very efficient and can work very well in practice. I implemented a very fast web server with very clean code using these coroutines. The cost of switching coroutines with this method is little more than the cost of a function call.

CZ has hygienic macros. The regular cpp (C preprocessor) macros are not hygenic and many people consider them hacky and unsafe to use. My CZ macros are safe, and somewhat more powerful than standard C macros. They can be used to neatly add new program control structures. I have plans to further develop the macro system in interesting ways.

I added automatic prototype and header generation, as I do not like having to repeat myself when copying prototypes to separate header files. I added support for the UNIX #! scripting syntax, and for cached executables, which means that CZ can be used like a scripting language without having to use a separate compile or make command, but the programs are only recompiled when something has been changed.

For CZ, I invented a neat approach to portability without conditional compilation directives. Platform-specific library fragments are automatically included from directories having the name of that platform or platform-category. This can work very well in practice, and helps to avoid the nightmare of conditional compilation, feature detection, and Autotools. Using this method, I was able easily to implement portable interfaces to features such as asynchronous IO multiplexing (aka select / poll).

The CZ library includes flexible error handling wrappers, inspired by W. Richard Stevens’ wrappers in his books on Unix Network Programming. If these wrappers are used, there is no need to check return values for error codes, and this makes the code much safer, as an error cannot accidentally be ignored.

CZ has several major faults, which I intend to correct at some point. Some of the syntax is poorly thought out, and I need to revisit it. I developed a fairly rich library to go with the language, including safer data structures, IO, networking, graphics, and sound. There are many nice features, but my CZ library is more prototype than a finished product, there are major omissions, and some features are misconceived or poorly implemented. The misfeatures should be weeded out for the time-being, or moved to an experimental section of the library.

I think that a good software library should come in two parts, the essential low-level APIs with the minimum necessary functionality, and a rich set of high-level convenience functions built on top of the minimal API. I need to clearly separate these two parts in order to avoid polluting the namespaces with all sorts of nonsense!

CZ is lacking a good modern system of symbol namespaces. I can look to Python for a great example. I need to maintain compatibility with C, and avoid ugly symbol encodings. I think I can come up with something that will alleviate the need to type anything like gtk_window_set_default_size, and yet maintain compatibility with the library in question. I want all the power of C, but it should be easy to use, even for children. It should be as easy as BASIC or Processing, a child should be able to write short graphical demos and the like, without stumbling over tricky syntax or obscure compile errors.

Here is an example of a simple CZ program which plots the Mandelbrot set fractal. I think that the program is fairly clear and easy to understand, although there is still some potential to improve and clarify the code.

#!/usr/local/bin/cz --
use b
use ccomplex

Main:
	num outside = 16, ox = -0.5, oy = 0, r = 1.5
	long i, max_i = 50, rb_i = 30
	space()
	uint32_t *px = pixel()  # CONFIGURE!
	num d = 2*r/h, x0 = ox-d*w_2, y0 = oy+d*h_2
	for(y, 0, h):
		cmplx c = x0 + (y0-d*y)*I
		repeat(w):
			cmplx w = c
			for i=0; i < max_i && cabs(w) < outside; ++i
				w = w*w + c
			*px++ = i < max_i ? rainbow(i*359 / rb_i % 360) : black
			c += d

I wrote a more elaborate variant of this program, which generates images like the one shown below. There are a few tricks used: continuous colouring, rainbow colours, and plotting the logarithm of the iteration count, which makes the plot appear less busy close to the black fractal proper. I sell some T-shirts and other products with these fractal designs online.

An image from the Mandelbrot set, generated by a fairly simple CZ program.

I am interested in graph programming, and have been for three decades since I was a teenager. By graph programming, I mean programming and modelling based on mathematical graphs or diagrams. I avoid the term visual programming, because there is no necessary reason that vision impaired folks could not use a graph programming language; a graph or diagram may be perceived, understood, and manipulated without having to see it.

Mathematics is something that naturally exists, outside time and independent of our universe. We humans discover mathematics, we do not invent or create it. One of my main ideas for graph programming is to represent a mathematical (or software) model in the simplest and most natural way, using relational operators. Elementary mathematics can be reduced to just a few such operators:

+add, subtract, disjoint union, zero
×multiply, divide, cartesian product, one
^power, root, logarithm
sin, cos, sin-1, cos-1, hypot, atan2
δdifferential, integral
a set of minimal relational operators for elementary math

I think that a language and notation based on these few operators (and similar) can be considerably simpler and more expressive than conventional math or programming languages.

CZ is for me a stepping-stone toward this goal of an expressive relational graph language. It is more pleasant for me to develop software tools in CZ than in C or another language.

Thanks for reading. I wrote this article during the process of applying to join Toptal, which appears to be a freelancing portal for top developers; and in response to this article on toptal: After All These Years, the World is Still Powered by C Programming.

My CZ project has been stalled for quite some time. I foolishly became discouraged after receiving some negative feedback. I now know that honest negative feedback should be valued as an opportunity to improve, and I intend to continue the project until it lacks glaring faults, and is useful for other people. If this project or this article interests you, please contact me and let me know. It is much more enjoyable to work on a project when other people are actively interested in it!

Gary PendergastWordPress Importers: Free (as in Speech)

Back at the start of this series, I listed four problems within the scope of the WordPress Importers that we needed to address. Three of them are largely technical problems, which I covered in previous posts. In wrapping up this series, I want to focus exclusively on the fourth problem, which has a philosophical side as well as a technical one — but that does not mean we cannot tackle it!

Problem Number 4

Some services work against their customers, and actively prevent site owners from controlling their own content.

Some services are merely inconvenient: they provide exports, but it often involves downloading a bunch of different files. Your CMS content is in one export, your store products are in another, your orders are in another, and your mailing list is in yet another. It’s not ideal, but they at least let you get a copy of your data.

However, there’s another class of services that actively work against their customers. It’s these services I want to focus on: the services that don’t provide any ability to export your content — effectively locking people in to using their platform. We could offer these folks an escape! The aim isn’t to necessarily make them use WordPress, it’s to give them a way out, if they want it. Whether they choose to use WordPress or not after that is immaterial (though I certainly hope they would, of course). The important part is freedom of choice.

It’s worth acknowledging that this is a different approach to how WordPress has historically operated in relation to other CMSes. We provide importers for many CMSes, but we previously haven’t written exporters. However, I don’t think this is a particularly large step: for CMSes that already provide exports, we’d continue to use those export files. This is focussed on the few services that try to lock their customers in.

Why Should WordPress Take This On?

There are several aspects to why we should focus on this.

First of all, it’s the the WordPress mission. Underpinning every part of WordPress is the simplest of statements:

Democratise Publishing

The freedom to build. The freedom to change. The freedom to share.

These freedoms are the pillars of a Free and Open Web, but they’re not invulnerable: at times, they need to be defended, and that needs people with the time and resources to offer a defence.

Which brings me to my second point: WordPress has the people who can offer that defence! The WordPress project has so many individuals working on it, from such a wide variety of backgrounds, we’re able to take on a vast array of projects that a smaller CMS just wouldn’t have the bandwidth for. That’s not to say that we can do everything, but when there’s a need to defend the entire ecosystem, we’re able to devote people to the cause.

Finally, it’s important to remember that WordPress doesn’t exist in a vacuum, we’re part of a broad ecosystem which can only exist through the web remaining open and free. By encouraging all CMSes to provide proper exports, and implementing them for those that don’t, we help keep our ecosystem healthy.

We have the ability to take on these challenges, but we have a responsibility that goes alongside. We can’t do it solely to benefit WordPress, we need to make that benefit available to the entire ecosystem. This is why it’s important to define a WordPress export schema, so that any CMS can make use of the export we produce, not just WordPress. If you’ll excuse the imagery for a moment, we can be the knight in shining armour that frees people — then gives them the choice of what they do with that freedom, without obligation.

How Can We Do It?

Moving on to the technical side of this problem, I can give you some good news: the answer is definitely not screen scraping. 😄 Scraping a site is fragile, impossible to transform into the full content, and provides an incomplete export of the site: anything that’s only available in the site dashboard can’t be obtained through scraping.

I’ve recently been experimenting with an alternative approach to solving this problem. Rather than trying to create something resembling a traditional exporter, it turns out that modern CMSes provide the tools we need, in the form of REST APIs. All we need to do is call the appropriate APIs, and collate the results. The fun part is that we can authenticate with these APIs as the site owner, by calling them from a browser extension! So, that’s what I’ve been experimenting with, and it’s showing a lot of promise.

If you’re interested in playing around with it, the experimental code is living in this repository. It’s a simple proof of concept, capable of exporting the text content of a blog on a Wix site, showing that we can make a smooth, comprehensive, easy-to-use exporter for any Wix site owner.

Screenshot of the "Free (as in Speech)" browser extension UI.

Clicking the export button starts a background script, which calls Wix’s REST APIs as the site owner, to get the original copy of the content. It then packages it up, and presents it as a WXR file to download.

Screenshot of a Firefox download dialog, showing a Wix site packaged up as a WXR file.

I’m really excited about how promising this experiment is. It can ultimately provide a full export of any Wix site, and we can add support for other CMS services that choose to artificially lock their customers in.

Where Can I Help?

If you’re a designer or developer who’s excited about working on something new, head on over to the repository and check out the open issues: if there’s something that isn’t already covered, feel free to open a new issue.

Since this is new ground for a WordPress project, both technically and philosophically, I’d love to hear more points of view. It’s being discussed in the WordPress Core Dev Chat this week, and you can also let me know what you think in the comments!

This post is part of a series, talking about the WordPress Importers, their history, where they are now, and where they could go in the future.

,

Gary PendergastWordPress Importers: Defining a Schema

While schemata are usually implemented using language-specific tools (eg, XML uses XML Schema, JSON uses JSON Schema), they largely use the same concepts when talking about data. This is rather helpful, we don’t need to make a decision on data formats before we can start thinking about how the data should be arranged.

Note: Since these concepts apply equally to all data formats, I’m using “WXR” in this post as shorthand for “the structured data section of whichever file format we ultimately use”, rather than specifically referring to the existing WXR format. 🙂

Why is a Schema Important?

It’s fair to ask why, if the WordPress Importers have survived this entire time without a formal schema, why would we need one now?

There are two major reasons why we haven’t needed one in the past:

  • WXR has remained largely unchanged in the last 10 years: there have been small additions or tweaks, but nothing significant. There’s been no need to keep track of changes.
  • WXR is currently very simple, with just a handful of basic elements. In a recent experiment, I was able to implement a JavaScript-based WXR generator in just a few days, entirely by referencing the Core implementation.

These reasons are also why it would help to implement a schema for the future:

  • As work on WXR proceeds, there will likely need to be substantial changes to what data is included: adding new fields, modifying existing fields, and removing redundant fields. Tracking these changes helps ensure any WXR implementations can stay in sync.
  • These changes will result in a more complex schema: relying on the source to re-implement it will become increasingly difficult and error-prone. Following Gutenberg’s lead, it’s likely that we’d want to provide official libraries in both PHP and JavaScript: keeping them in sync is best done from a source schema, rather than having one implementation copy the other.

Taking the time to plan out a schema now gives us a solid base to work from, and it allows for future changes to happen in a reliable fashion.

WXR for all of WordPress

With a well defined schema, we can start to expand what data will be included in a WXR file.

Media

Interestingly, many of the challenges around media files are less to do with WXR, and more to do with importer capabilities. The biggest headache is retrieving the actual files, which the importer currently handles by trying to retrieve the file from the remote server, as defined in the wp:attachment_url node. In context, this behaviour is understandable: 10+ years ago, personal internet connections were too slow to be moving media around, it was better to have the servers talk to each other. It’s a useful mechanism that we should keep as a fallback, but the more reliable solution is to include the media file with the export.

Plugins and Themes

There are two parts to plugins and themes: the code, and the content. Modern WordPress sites require plugins to function, and most are customised to suit their particular theme.

For exporting the code, I wonder if a tiered solution could be applied:

  • Anything from WordPress.org would just need their slug, since they can be re-downloaded during import. Particularly as WordPress continues to move towards an auto-updated future, modified versions of plugins and themes are explicitly not supported.
  • Third party plugins and themes would be given a filter to use, where they can provide a download URL that can be included in the export file.
  • Third party plugins/themes that don’t provide a download URL would either need to be skipped, or zipped up and included in the export file.

For exporting the content, WXR already includes custom post types, but doesn’t include custom settings, or custom tables. The former should be included automatically, and the latter would likely be handled by an appropriate action for the plugin to hook into.

Settings

There are a currently handful of special settings that are exported, but (as I just noted, particularly with plugins and themes being exported) this would likely need to be expanded to included most items in wp_options.

Users

Currently, the bare minimum information about users who’ve authored a post is included in the export. This would need to be expanded to include more user information, as well as users who aren’t post authors.

WXR for parts of WordPress

The modern use case for importers isn’t just to handle a full site, but to handle keeping sites in sync. For example, most news organisations will have a staging site (or even several layers of staging!) which is synchronised to production.

While it’s well outside the scope of this project to directly handle every one of these use cases, we should be able to provide the framework for organisations to build reliable platforms on. Exports should be repeatable, objects in the export should have unique identifiers, and the importer should be able to handle any subset of WXR.

WXR Beyond WordPress

Up until this point, we’ve really been talking about WordPress→WordPress migrations, but I think WXR is a useful format beyond that. Instead of just containing direct exports of the data from particular plugins, we could also allow it to contain “types” of data. This turns WXR into an intermediary language, exports can be created from any source, and imported into WordPress.

Let’s consider an example. Say we create a tool that can export a Shopify, Wix, or GoDaddy site to WXR, how would we represent an online store in the WXR file? We don’t want to export in the format that any particular plugin would use, since a WordPress Core tool shouldn’t be advantaging one plugin over others.

Instead, it would be better if we could format the data in a platform-agnostic way, which plugins could then implement support for. As luck would have it, Schema.org provides exactly the kind of data structure we could use here. It’s been actively maintained for nearly nine years, it supports a wide variety of data types, and is intentionally platform-agnostic.

Gazing into my crystal ball for a moment, I can certainly imagine a future where plugins could implement and declare support for importing certain data types. When handling such an import (assuming one of those plugins wasn’t already installed), the WordPress Importer could offer them as options during the import process. This kind of seamless integration allows WordPress to show that it offers the same kind of fully-featured site building experience that modern CMS services do.

Of course, reality is never quite as simple as crystal balls and magic wands make them out to be. We have to contend with services that provide incomplete or fragmented exports, and there are even services that deliberately don’t provide exports at all. In the next post, I’ll be writing about why we should address this problem, and how we might be able to go about it.

This post is part of a series, talking about the WordPress Importers, their history, where they are now, and where they could go in the future.

,

Gary PendergastWordPress Importers: Getting Our House in Order

The previous post talked about the broad problems we need to tackle to bring our importers up to speed, making them available for everyone to use.

In this post, I’m going to focus on what we could do with the existing technology, in order to give us the best possible framework going forward.

A Reliable Base

Importers are an interesting technical problem. Much like you’d expect from any backup/restore code, importers need to be extremely reliable. They need to comfortable handle all sorts of unusual data, and they need to keep it all safe. Particularly considering their age, the WordPress Importers do a remarkably good job of handling most content you can throw at it.

However, modern development practices have evolved and improved since the importers were first written, and we should certainly be making use of such practices, when they fit with our requirements.

For building reliable software that we expect to largely run by itself, a variety of comprehensive automated testing is critical. This ensures we can confidently take on the broader issues, safe in the knowledge that we have a reliable base to work from.

Testing must be the first item on this list. A variety of automated testing gives us confidence that changes are safe, and that the code can continue to be maintained in the future.

Data formats must be well defined. While this is useful for ensuring data can be handled in a predictable fashion, it’s also a very clear demonstration of our commitment to data freedom.

APIs for creating or extending importers should be straightforward for hooking into.

Performance Isn’t an Optional Extra

With sites constantly growing in size (and with the export files potentially gaining a heap of extra data), we need to care about the performance of the importers.

Luckily, there’s already been some substantial work done on this front:

There are other groups in the WordPress world who’ve made performance improvements in their own tools: gathering all of that experience is a relatively quick way to bring in production-tested improvements.

The WXR Format

It’s worth talking about the WXR format itself, and determining whether it’s the best option for handling exports into the future. XML-based formats are largely viewed as a relic of days gone past, so (if we were to completely ignore backwards compatibility for a moment) is there a modern data format that would work better?

The short answer… kind of. 🙂

XML is actually well suited to this use case, and (particularly when looking at performance improvements) is the only data format for which PHP comes with a built-in streaming parser.

That said, WXR is basically an extension of the RSS format: as we add more data to the file that clearly doesn’t belong in RSS, there is likely an argument for defining an entirely WordPress-focused schema.

Alternative Formats

It’s important to consider what the priorities are for our export format, which will help guide any decision we make. So, I’d like to suggest the following priorities (in approximate priority order):

  • PHP Support: The format should be natively supported in PHP, thought it is still workable if we need to ship an additional library.
  • Performant: Particularly when looking at very large exports, it should be processed as quickly as possible, using minimal RAM.
  • Supports Binary Files: The first comments on my previous post asked about media support, we clearly should be treating it as a first-class citizen.
  • Standards Based: Is the format based on a documented standard? (Another way to ask this: are there multiple different implementations of the format? Do those implementations all function the same?
  • Backward Compatible: Can the format be used by existing tools with no changes, or minimal changes?
  • Self Descriptive: Does the format include information about what data you’re currently looking at, or do you need to refer to a schema?
  • Human Readable: Can the file be opened and read in a text editor?

Given these priorities, what are some options?

WXR (XML-based)

Either the RSS-based schema that we already use, or a custom-defined XML schema, the arguments for this format are pretty well known.

One argument that hasn’t been well covered is how there’s a definite trade-off when it comes to supporting binary files. Currently, the importer tries to scrape the media file from the original source, which is not particularly reliable. So, if we were to look at including media files in the WXR file, the best option for storing them is to base64 encode them. Unfortunately, that would have a serious effect on performance, as well as readability: adding huge base64 strings would make even the smallest exports impossible to read.

Either way, this option would be mostly backwards compatible, though some tools may require a bit of reworking if we were to substantial change the schema.

WXR (ZIP-based)

To address the issues with media files, an alternative option might be to follow the path that Microsoft Word and OpenOffice use: put the text content in an XML file, put the binary content into folders, and compress the whole thing.

This addresses the performance and binary support problems, but is initially worse for readability: if you don’t know that it’s a ZIP file, you can’t read it in a text editor. Once you unzip it, however, it does become quite readable, and has the same level of backwards compatibility as the XML-based format.

JSON

JSON could work as a replacement for XML in both of the above formats, with one additional caveat: there is no streaming JSON parser built in to PHP. There are 3rd party libraries available, but given the documented differences between JSON parsers, I would be wary about using one library to produce the JSON, and another to parse it.

This format largely wouldn’t be backwards compatible, though tools which rely on the export file being plain text (eg, command line tools to do broad search-and-replaces on the file) can be modified relatively easily.

There are additional subjective arguments (both for and against) the readability of JSON vs XML, but I’m not sure there’s anything to them beyond personal preference.

SQLite

The SQLite team wrote an interesting (indirect) argument on this topic: OpenOffice uses a ZIP-based format for storing documents, the SQLite team argued that there would be benefits (particularly around performance and reliability) for OpenOffice to switch to SQLite.

They key issues that I see are:

  • SQLite is included in PHP, but not enabled by default on Windows.
  • While the SQLite team have a strong commitment to providing long-term support, SQLite is not a standard, and the only implementation is the one provided by the SQLite team.
  • This option is not backwards compatible at all.

FlatBuffers

FlatBuffers is an interesting comparison, since it’s a data format focussed entirely on speed. The down side of this focus is that it requires a defined schema to read the data. Much like SQLite, the only standard for FlatBuffers is the implementation. Unlike SQLite, FlatBuffers has made no commitments to providing long-term support.

WXR (XML-based)WXR (ZIP-based)JSONSQLiteFlatBuffers
Works in PHP?✅✅⚠⚠⚠
Performant?⚠✅⚠✅✅
Supports Binary Files?⚠✅⚠✅✅
Standards Based?✅✅✅⚠ / ��
Backwards Compatible?⚠⚠���
Self Descriptive?✅✅✅✅�
Readable?✅⚠ / �✅��

As with any decision, this is a matter of trade-offs. I’m certainly interested in hearing additional perspectives on these options, or thoughts on options that I haven’t considered.

Regardless of which particular format we choose for storing WordPress exports, every format should have (or in the case of FlatBuffers, requires) a schema. We can talk about schemata without going into implementation details, so I’ll be writing about that in the next post.

This post is part of a series, talking about the WordPress Importers, their history, where they are now, and where they could go in the future.

Gary PendergastWordPress Importers: Stating the Problem

It’s time to focus on the WordPress Importers.

I’m not talking about tidying them up, or improve performance, or fixing some bugs, though these are certainly things that should happen. Instead, we need to consider their purpose, how they fit as a driver of WordPress’ commitment to Open Source, and how they can be a key element in helping to keep the Internet Open and Free.

The History

The WordPress Importers are arguably the key driver to WordPress’ early success. Before the importer plugins existed (before WordPress even supported plugins!) there were a handful of import-*.php scripts in the wp-admin directory that could be used to import blogs from other blogging platforms. When other platforms fell out of favour, WordPress already had an importer ready for people to move their site over. One of the most notable instances was in 2004, when Moveable Type changed their license and prices, suddenly requiring personal blog authors to pay for something that had previously been free. WordPress was fortunate enough to be in the right place at the right time: many of WordPress’ earliest users came from Moveable Type.

As time went on, WordPress became well known in its own right. Growth relied less on people wanting to switch from another provider, and more on people choosing to start their site with WordPress. For practical reasons, the importers were moved out of WordPress Core, and into their own plugins. Since then, they’ve largely been in maintenance mode: bugs are fixed when they come up, but since export formats rarely change, they’ve just continued to work for all these years.

An unfortunate side effect of this, however, is that new importers are rarely written. While a new breed of services have sprung up over the years, the WordPress importers haven’t kept up.

The New Services

There are many new CMS services that have cropped up in recent years, and we don’t have importers for any of them. WordPress.com has a few extra ones written, but they’ve been built on the WordPress.com infrastructure out of necessity.

You see, we’ve always assumed that other CMSes will provide some sort of export file that we can use to import into WordPress. That isn’t always the case, however. Some services (notable, Wix and GoDaddy Website Builder) deliberately don’t allow you to export your own content. Other services provide incomplete or fragmented exports, needlessly forcing stress upon site owners who want to use their own content outside of that service.

To work around this, WordPress.com has implemented importers that effectively scrape the site: while this has worked to some degree, it does require regular maintenance, and the importer has to do a lot of guessing about how the content should be transformed. This is clearly not a solution that would be maintainable as a plugin.

Problem Number 4

Some services work against their customers, and actively prevent site owners from controlling their own content.

This strikes at the heart of the WordPress Bill of Rights. WordPress is built with fundamental freedoms in mind: all of those freedoms point to owning your content, and being able to make use of it in any form you like. When a CMS actively works against providing such freedom to their community, I would argue that we have an obligation to help that community out.

A Variety of Content

It’s worth discussing how, when starting a modern CMS service, the bar for success is very high. You can’t get away with just providing a basic CMS: you need to provide all the options. Blogs, eCommerce, mailing lists, forums, themes, polls, statistics, contact forms, integrations, embeds, the list goes on. The closest comparison to modern CMS services is… the entire WordPress ecosystem: built on WordPress core, but with the myriad of plugins and themes available, along with the variety of services offered by a huge array of companies.

So, when we talk about the importers, we need to consider how they’ll be used.

Problem Number 3

To import from a modern CMS service into WordPress, your importer needs to map from service features to WordPress plugins.

Getting Our Own House In Order

Some of these problems don’t just apply to new services, however.

Out of the box, WordPress exports to WXR (WordPress eXtended RSS) files: an XML file that contains the content of the site. Back when WXR was first created, this was all you really needed, but much like the rest of the WordPress importers, it hasn’t kept up with the times. A modern WordPress site isn’t just the sum of its content: a WordPress site has plugins and themes. It has various options configured, it has huge quantities of media, it has masses of text content, far more than the first WordPress sites ever had.

Problem Number 2

WXR doesn’t contain a full export of a WordPress site.

In my view, WXR is a solid format for handling exports. An XML-based system is quite capable of containing all forms of content, so it’s reasonable that we could expand the WXR format to contain the entire site.

Built for the Future

If there’s one thing we can learn from the history of the WordPress importers, it’s that maintenance will potentially be sporadic. Importers are unlikely to receive the same attention that the broader WordPress Core project does, owners may come and go. An importer will get attention if it breaks, of course, but it otherwise may go months or years without changing.

Problem Number 1

We can’t depend on regular importer maintenance in the future.

It’s quite possible to build code that will be running in 10+ years: we see examples all across the WordPress ecosystem. Doing it in a reliable fashion needs to be a deliberate choice, however.

What’s Next?

Having worked our way down from the larger philosophical reasons for the importers, to some of the more technically-oriented implementation problems; I’d like to work our way back out again, focussing on each problem individually. In the following posts, I’ll start laying out how I think we can bring our importers up to speed, prepare them for the future, and make them available for everyone.

This post is part of a series, talking about the WordPress Importers, their history, where they are now, and where they could go in the future.

,

Glen TurnerCompiling and installing software for the uBITX v6 QRP amateur radio transciever

The uBITX uses an Arduino internally. This article describes how to update its software.

Required hardware

The connector on the back is a Mini-B USB connector, so you'll need a "Mini-B to A" USB cable. This is not the same cable as used with older Android smartphones. The Mini-B connector was used with a lot of cameras a decade ago.

You'll also need a computer. I use a laptop with Fedora Linux installed.

Required software for software development

In Fedora all the required software is installed with sudo dnf install arduino git. Add yourself to the users and lock groups with sudo usermod -a -G users,lock $USER (on Debian-style systems use sudo usermod -a -G dialout,lock $USER). You'll need to log out and log in again for that to have an effect (if you want to see which groups you are already in, then use the id command).

Run arduino as your ordinary non-root user to create the directories used by the Arduino IDE. You can quit the IDE once it starts.

Obtain the uBITX software

$ cd ~/Arduino
$ git clone https://github.com/afarhan/ubitxv6.git ubitx_v6.1_code

Connect the uBITX to your computer

Plug in the USB cable and turn on the radio. Running dmesg will show the Arduino appearing as a "USB serial" device:

usb 1-1: new full-speed USB device number 6 using xhci_hcd
usb 1-1: New USB device found, idVendor=1a86, idProduct=7523, bcdDevice= 2.64
usb 1-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
usb 1-1: Product: USB Serial
usbcore: registered new interface driver ch341
usbserial: USB Serial support registered for ch341-uart
ch341 1-1:1.0: ch341-uart converter detected
usb 1-1: ch341-uart converter now attached to ttyUSB1

If you want more information about the USB device then use:

$ lsusb -d 1a86:7523
Bus 001 Device 006: ID 1a86:7523 QinHeng Electronics CH340 serial converter


comment count unavailable comments

,

Jan SchmidtRift CV1 – Adventures in Kalman filtering Part 2

In the last post I had started implementing an Unscented Kalman Filter for position and orientation tracking in OpenHMD. Over the Christmas break, I continued that work.

A Quick Recap

When reading below, keep in mind that the goal of the filtering code I’m writing is to combine 2 sources of information for tracking the headset and controllers.

The first piece of information is acceleration and rotation data from the IMU on each device, and the second is observations of the device position and orientation from 1 or more camera sensors.

The IMU motion data drifts quickly (at least for position tracking) and can’t tell which way the device is facing (yaw, but can detect gravity and get pitch/roll).

The camera observations can tell exactly where each device is, but arrive at a much lower rate (52Hz vs 500/1000Hz) and can take a long time to process (hundreds of milliseconds) to analyse to acquire or re-acquire a lock on the tracked device(s).

The goal is to acquire tracking lock, then use the motion data to predict the motion closely enough that we always hit the ‘fast path’ of vision analysis. The key here is closely enough – the more closely the filter can track and predict the motion of devices between camera frames, the better.

Integration in OpenHMD

When I wrote the last post, I had the filter running as a standalone application, processing motion trace data collected by instrumenting a running OpenHMD app and moving my headset and controllers around. That’s a really good way to work, because it lets me run modifications on the same data set and see what changed.

However, the motion traces were captured using the current fusion/prediction code, which frequently loses tracking lock when the devices move – leading to big gaps in the camera observations and more interpolation for the filter.

By integrating the Kalman filter into OpenHMD, the predictions are improved leading to generally much better results. Here’s one trace of me moving the headset around reasonably vigourously with no tracking loss at all.

Headset motion capture trace

If it worked this well all the time, I’d be ecstatic! The predicted position matched the observed position closely enough for every frame for the computer vision to match poses and track perfectly. Unfortunately, this doesn’t happen every time yet, and definitely not with the controllers – although I think the latter largely comes down to the current computer vision having more troubler matching controller poses. They have fewer LEDs to match against compared to the headset, and the LEDs are generally more side-on to a front-facing camera.

Taking a closer look at a portion of that trace, the drift between camera frames when the position is interpolated using the IMU readings is clear.

Headset motion capture – zoomed in view

This is really good. Most of the time, the drift between frames is within 1-2mm. The computer vision can only match the pose of the devices to within a pixel or two – so the observed jitter can also come from the pose extraction, not the filtering.

The worst tracking is again on the Z axis – distance from the camera in this case. Again, that makes sense – with a single camera matching LED blobs, distance is the most uncertain part of the extracted pose.

Losing Track

The trace above is good – the computer vision spots the headset and then the filtering + computer vision track it at all times. That isn’t always the case – the prediction goes wrong, or the computer vision fails to match (it’s definitely still far from perfect). When that happens, it needs to do a full pose search to reacquire the device, and there’s a big gap until the next pose report is available.

That looks more like this

Headset motion capture trace with tracking errors

This trace has 2 kinds of errors – gaps in the observed position timeline during full pose searches and erroneous position reports where the computer vision matched things incorrectly.

Fixing the errors in position reports will require improving the computer vision algorithm and would fix most of the plot above. Outlier rejection is one approach to investigate on that front.

Latency Compensation

There is inherent delay involved in processing of the camera observations. Every 19.2ms, the headset emits a radio signal that triggers each camera to capture a frame. At the same time, the headset and controller IR LEDS light up brightly to create the light constellation being tracked. After the frame is captured, it is delivered over USB over the next 18ms or so and then submitted for vision analysis. In the fast case where we’re already tracking the device the computer vision is complete in a millisecond or so. In the slow case, it’s much longer.

Overall, that means that there’s at least a 20ms offset between when the devices are observed and when the position information is available for use. In the plot above, this delay is ignored and position reports are fed into the filter when they are available. In the worst case, that means the filter is being told where the headset was hundreds of milliseconds earlier.

To compensate for that delay, I implemented a mechanism in the filter where it keeps extra position and orientation entries in the state that can be used to retroactively apply the position observations.

The way that works is to make a prediction of the position and orientation of the device at the moment the camera frame is captured and copy that prediction into the extra state variable. After that, it continues integrating IMU data as it becomes available while keeping the auxilliary state constant.

When a the camera frame analysis is complete, that delayed measurement is matched against the stored position and orientation prediction in the state and the error used to correct the overall filter. The cool thing is that in the intervening time, the filter covariance matrix has been building up the right correction terms to adjust the current position and orientation.

Here’s a good example of the difference:

Before: Position filtering with no latency compensation
After: Latency-compensated position reports

Notice how most of the disconnected segments have now slotted back into position in the timeline. The ones that haven’t can either be attributed to incorrect pose extraction in the compute vision, or to not having enough auxilliary state slots for all the concurrent frames.

At any given moment, there can be a camera frame being analysed, one arriving over USB, and one awaiting “long term” analysis. The filter needs to track an auxilliary state variable for each frame that we expect to get pose information from later, so I implemented a slot allocation system and multiple slots.

The downside is that each slot adds 6 variables (3 position and 3 orientation) to the covariance matrix on top of the 18 base variables. Because the covariance matrix is square, the size grows quadratically with new variables. 5 new slots means 30 new variables – leading to a 48 x 48 covariance matrix instead of 18 x 18. That is a 7-fold increase in the size of the matrix (48 x 48 = 2304 vs 18 x 18 = 324) and unfortunately about a 10x slow-down in the filter run-time.

At that point, even after some optimisation and vectorisation on the matrix operations, the filter can only run about 3x real-time, which is too slow. Using fewer slots is quicker, but allows for fewer outstanding frames. With 3 slots, the slow-down is only about 2x.

There are some other possible approaches to this problem:

  • Running the filtering delayed, only integrating IMU reports once the camera report is available. This has the disadvantage of not reporting the most up-to-date estimate of the user pose, which isn’t great for an interactive VR system.
  • Keeping around IMU reports and rewinding / replaying the filter for late camera observations. This limits the overall increase in filter CPU usage to double (since we at most replay every observation twice), but potentially with large bursts when hundreds of IMU readings need replaying.
  • It might be possible to only keep 2 “full” delayed measurement slots with both position and orientation, and to keep some position-only slots for others. The orientation of the headset tends to drift much more slowly than position does, so when there’s a big gap in the tracking it would be more important to be able to correct the position estimate. Orientation is likely to still be close to correct.
  • Further optimisation in the filter implementation. I was hoping to keep everything dependency-free, so the filter implementation uses my own naive 2D matrix code, which only implements the features needed for the filter. A more sophisticated matrix library might perform better – but it’s hard to say without doing some testing on that front.

Controllers

So far in this post, I’ve only talked about the headset tracking and not mentioned controllers. The controllers are considerably harder to track right now, but most of the blame for that is in the computer vision part. Each controller has fewer LEDs than the headset, fewer are visible at any given moment, and they often aren’t pointing at the camera front-on.

Oculus Camera view of headset and left controller.

This screenshot is a prime example. The controller is the cluster of lights at the top of the image, and the headset is lower left. The computer vision has gotten confused and thinks the controller is the ring of random blue crosses near the headset. It corrected itself a moment later, but those false readings make life very hard for the filtering.

Position tracking of left controller with lots of tracking loss.

Here’s a typical example of the controller tracking right now. There are some very promising portions of good tracking, but they are interspersed with bursts of tracking losses, and wild drifting from the computer vision giving wrong poses – leading to the filter predicting incorrect acceleration and hence cascaded tracking losses. Particularly (again) on the Z axis.

Timing Improvements

One of the problems I was looking at in my last post is variability in the arrival timing of the various USB streams (Headset reports, Controller reports, camera frames). I improved things in OpenHMD on that front, to use timestamps from the devices everywhere (removing USB timing jitter from the inter-sample time).

There are still potential problems in when IMU reports from controllers get updated in the filters vs the camera frames. That can be on the order of 2-4ms jitter. Time will tell how big a problem that will be – after the other bigger tracking problems are resolved.

Sponsorships

All the work that I’m doing implementing this positional tracking is a combination of my free time, hours contributed by my employer Centricular and contributions from people via Github Sponsorships. If you’d like to help me spend more hours on this and fewer on other paying work, I appreciate any contributions immensely!

Next Steps

The next things on my todo list are:

  • Integrate the delayed-observation processing into OpenHMD (at the moment it is only in my standalone simulator).
  • Improve the filter code structure – this is my first kalman filter and there are some implementation decisions I’d like to revisit.
  • Publish the UKF branch for other people to try.
  • Circle back to the computer vision and look at ways to improve the pose extraction and better reject outlying / erroneous poses, especially for the controllers.
  • Think more about how to best handle / schedule analysis of frames from multiple cameras. At the moment each camera operates as a separate entity, capturing frames and analysing them in threads without considering what is happening in other cameras. That means any camera that can’t see a particular device starts doing full pose searches – which might be unnecessary if another camera still has a good view of the device. Coordinating those analyses across cameras could yield better CPU consumption, and let the filter retain fewer delayed observation slots.

,

Tim SerongScope Creep

On December 22, I decided to brew an oatmeal stout (5kg Gladfield ale malt, 250g dark chocolate malt, 250g light chocolate malt, 250g dark crystal malt, 500g rolled oats, 150g rice hulls to stop the mash sticking, 25g Pride of Ringwood hops, Safale US-05 yeast). This all takes a good few hours to do the mash and the boil and everything, so while that was underway I thought it’d be a good opportunity to remove a crappy old cupboard from the laundry, so I could put our nice Miele upright freezer in there, where it’d be closer to the kitchen (the freezer is presently in a room at the other end of the house).

The cupboard was reasonably easy to rip out, but behind it was a mouldy and unexpectedly bright yellow wall with an ugly gap at the top where whoever installed it had removed the existing cornice.

Underneath the bottom half of the cupboard, I discovered not the cork tiles which cover the rest of the floor, but a layer of horrific faux-tile linoleum. Plus, more mould. No way was I going to put the freezer on top of that.

So, up came the floor covering, back to nice hardwood boards.

Of course, the sink had to come out too, to remove the flooring from under its cabinet, and that meant pulling the splashback tiles (they had ugly screw holes in them anyway from a shelf that had been bracketed up on top of them previously).

Removing the tiles meant replacing a couple of sections of wall.

Also, we still needed to be able to use the washing machine through all this, so I knocked up a temporary sink support.

New cornice went in.

The rest of the plastering was completed and a ceiling fan installed.

Waterproofing membrane was applied where new tiles will go around a new sink.

I removed the hideous old aluminium backed weather stripping from around the exterior door and plastered up the exposed groove.

We still need to paint everything, get the new sink installed, do the tiling work and install new taps.

As for the oatmeal stout, I bottled that on January 2. From a sample taken at the time, it should be excellent, but right now still needs to carbonate and mature.

Stewart SmithPhotos from Taiwan

A few years ago we went to Taiwan. I managed to capture some random bits of the city on film (and also some shots on my then phone, a Google Pixel). I find the different style of art on the streets around the world to be fascinating, and Taiwan had some good examples.

I’ve really enjoyed shooting Kodak E100VS film over the years, and some of my last rolls were shot in Taiwan. It’s a film that unfortunately is not made anymore, but at least we have a new Ektachrome to have fun with now.

Words for our time: “Where there is democracy, equality and freedom can exist; without democracy, equality and freedom are merely empty words”.

This is, of course, only a small number of the total photos I took there. I’d really recommend a trip to Taiwan, and I look forward to going back there some day.

,

Colin CharlesCiao, 2020

Another year comes to a close, and this is the 4th year running I’m in Kuala Lumpur — 2017, 2018, 2019, and 2020… Wow. Maybe the biggest difference is that I’ve been in Malaysia for 306 days, thanks to the novel coronavirus. I have never spent this much time in Malaysia, in my entire life… I want to say KL, but I’ve managed to zip my way around to Kuantan (a lot), Penang, and Malacca. I can’t believe I flew back on February 29 2020 from Tokyo, and never got on a plane again! What a grounded globalist I’ve become.

My travel stats are of course, pretty dismal. 39 days out of the country. Apparently I did a total of 13 trips, 92 days of travel (I don’t know if all my local trips are counted frankly), 60,766km, 17 cities, and still 7 countries :) I don’t even want to compare to what it was like in 2019.

I ended that by saying, “I welcome 2020 with arms wide open.”. I’m not so sure how I feel about 2020. There is life beyond travel. COVID and our reaction to it, really worries me.

KL has some pretty good food. Kuantan has some pretty good people. While in KL, I visited a spin studio at least once per day. I did a total of 272 spin classes over 366 days! Not to forget there was 56 days of complete lockdown, and studios didn’t open till about maybe mid-June… Sure I did do some spin in London and Paris too, but the bulk of all this happened while I was here in KL.

I became reasonably friendlier, I became vulnerable, and like every time you do that, you’re chances of happiness and getting hurt probably straddle 50:50. Madonna – The Power of Good-bye can be apt.

This is not to say I didn’t enjoy 2020. Glass half full. I really did. Carpe diem. Simplicity is best. If you can follow KISS principles in engineering, why would you pour your entire thought process out and overwhelm the other party?

Anyway, I still look forward to 2021, with wide open arms, and while I really do think the COVID mess isn’t going away and things are going to be worse for many, I will still be focused on the most positive aspects of 2021. And I’ll work on being my old self again ;-)

I also ended the year with a haircut (number 1/0.5 on the sides) on Monday 28 December 2020. Somewhat of an experiment (does CoQ10 help speed up hair growth?) but also somewhat of a reaction to saying goodbye to December 2020.

The post Ciao, 2020 first appeared on Colin Charles Agenda.

,

Stewart SmithTwo Photos from Healseville Sanctuary

If you’re near Melbourne, you should go to Healseville Sanctuary and enjoy the Australian native animals. I’ve been a number of times over the years, and here’s a couple of photos from a (relatively, as in, the last couple of years) trip.

Leah trying to photograph a much too close bird
Koalas seem to always look like they’ve just woken up. I’m pretty convinced this one just had.

Stewart SmithPhotos from Adelaide

Some shots on Kodak Portra 400 from Adelaide. These would have been shot with my Nikon F80 35mm body, I think all with the 50mm lens. These are all pre-pandemic, and I haven’t gone and looked up when exactly. I’m just catching up on scanning some negatives.

,

Glen TurnerBlocking a USB device

udev can be used to block a USB device (or even an entire class of devices, such as USB storage). Add a file /etc/udev/rules.d/99-local-blacklist.rules containing:

SUBSYSTEM=="usb", ATTRS{idVendor}=="0123", ATTRS{idProduct}=="4567", ATTR{authorized}="0"


comment count unavailable comments

,

Hamish TaylorWattlebird feeding

While I hope to update this site again soon, here’s a photo I captured over the weekend in my back yard. The red flowering plant is attracting wattlebirds and honey-eaters. This wattlebird stayed still long enough for me to take this shot. After a little bit of editing, I think it has turned out rather well.

Photo taken with: Canon 7D Mark II & Canon 55-250mm lens.

Edited in Lightroom and Photoshop (to remove a sun glare spot off the eye).

Wattlebird feeding

Gary PendergastMore than 280 characters

It’s hard to be nuanced in 280 characters.

The Twitter character limit is a major factor of what can make it so much fun to use: you can read, publish, and interact, in extremely short, digestible chunks. But, it doesn’t fit every topic, ever time. Sometimes you want to talk about complex topics, having honest, thoughtful discussions. In an environment that encourages hot takes, however, it’s often easier to just avoid having those discussions. I can’t blame people for doing that, either: I find myself taking extended breaks from Twitter, as it can easily become overwhelming.

For me, the exception is Twitter threads.

Twitter threads encourage nuance and creativity.

Creative masterpieces like this Choose Your Own Adventure are not just possible, they rely on Twitter threads being the way they are.

Publishing a short essay about your experiences in your job can bring attention to inequality.

And Tumblr screenshot threads are always fun to read, even when they take a turn for the epic (over 4000 tweets in this thread, and it isn’t slowing down!)

Everyone can think of threads that they’ve loved reading.

My point is, threads are wildly underused on Twitter. I think I big part of that is the UI for writing threads: while it’s suited to writing a thread as a series of related tweet-sized chunks, it doesn’t lend itself to writing, revising, and editing anything more complex.

To help make this easier, I’ve been working on a tool that will help you publish an entire post to Twitter from your WordPress site, as a thread. It takes care of transforming your post into Twitter-friendly content, you can just… write. 🙂

It doesn’t just handle the tweet embeds from earlier in the thread: it handles handle uploading and attaching any images and videos you’ve included in your post.

All sorts of embeds work, too. 😉

It’ll be coming in Jetpack 9.0 (due out October 6), but you can try it now in the latest Jetpack Beta! Check it out and tell me what you think. 🙂

This might not fix all of Twitter’s problems, but I hope it’ll help you enjoy reading and writing on Twitter a little more. 💖

,

Glen TurnerConverting MPEG-TS to, well, MPEG

Digital TV uses MPEG Transport Stream, which is a container for video designed for lossy transmission, such as radio. To save CPU cycles, Personal Video Records often save the MPEG-TS stream directly to disk. The more usual MPEG is technically MPEG Program Stream, which is designed for lossless transmission, such as storage on a disk.

Since these are a container formats, it should be possible to losslessly and quickly re-code from MPEG-TS to MPEG-PS.

ffmpeg -ss "${STARTTIME}" -to "${DURATION}" -i "${FILENAME}" -ignore_unknown -map 0 -map -0:2 -c copy "${FILENAME}.mpeg"


comment count unavailable comments

,

Chris NeugebauerTalk Notes: Practicality Beats Purity: The Zen Of Python’s Escape Hatch?

I gave the talk Practicality Beats Purity: The Zen of Python’s Escape Hatch as part of PyConline AU 2020, the very online replacement for PyCon AU this year. In that talk, I included a few interesting links code samples which you may be interested in:

@apply

def apply(transform):

    def __decorator__(using_this):
        return transform(using_this)

    return __decorator__


numbers = [1, 2, 3, 4, 5]

@apply(lambda f: list(map(f, numbers)))
def squares(i):
  return i * i

print(list(squares))

# prints: [1, 4, 9, 16, 25]

Init.java

public class Init {
  public static void main(String[] args) {
    System.out.println("Hello, World!")
  }
}

@switch and @case

__NOT_A_MATCHER__ = object()
__MATCHER_SORT_KEY__ = 0

def switch(cls):

    inst = cls()
    methods = []

    for attr in dir(inst):
        method = getattr(inst, attr)
        matcher = getattr(method, "__matcher__", __NOT_A_MATCHER__)

        if matcher == __NOT_A_MATCHER__:
            continue

        methods.append(method)

    methods.sort(key = lambda i: i.__matcher_sort_key__)

    for method in methods:
        matches = method.__matcher__()
        if matches:
            return method()

    raise ValueError(f"No matcher matches value {test_value}")

def case(matcher):

    def __decorator__(f):
        global __MATCHER_SORT_KEY__

        f.__matcher__ = matcher
        f.__matcher_sort_key__ = __MATCHER_SORT_KEY__
        __MATCHER_SORT_KEY__ += 1
        return f

    return __decorator__



if __name__ == "__main__":
    for i in range(100):

        @switch
        class FizzBuzz:

            @case(lambda: i % 15 == 0)
            def fizzbuzz(self):
                return "fizzbuzz"

            @case(lambda: i % 3 == 0)
            def fizz(self):
                return "fizz"

            @case(lambda: i % 5 == 0)
            def buzz(self):
                return "buzz"

            @case(lambda: True)
            def default(self):
                return "-"

        print(f"{i} {FizzBuzz}")

,

Colin CharlesLinks on Rona #2

This was easily a late April 2020 roundup, stuck in BBEdit, which may still be vaguely relevant.

The post Links on Rona #2 first appeared on Colin Charles Agenda.

,

Craig SandersFuck Grey Text

fuck grey text on white backgrounds
fuck grey text on black backgrounds
fuck thin, spindly fonts
fuck 10px text
fuck any size of anything in px
fuck font-weight 300
fuck unreadable web pages
fuck themes that implement this unreadable idiocy
fuck sites that don’t work without javascript
fuck reactjs and everything like it

thank fuck for Stylus. and uBlock Origin. and uMatrix.

Fuck Grey Text is a post from: Errata

,

Hamish TaylorBlog: A new beginning

Earlier today I launched this site. It is the result of a lot of work over the past few weeks. It began as an idea to publicise some of my photos, and morphed into the site you see now, including a store and blog that I’ve named “Photekgraddft”.

In the weirdly named blog, I want to talk about photography, the stories behind some of my more interesting shots, the gear and software I use, my technology career, my recent ADHD diagnosis and many other things.

This scares me quite a lot. I’ve never really put myself out onto the internet before. If you Google me, you’re not going to find anything much. Google Images has no photos of me. I’ve always liked it that way. Until now.

ADHD’ers are sometimes known for “oversharing”, one of the side-effects of the inability to regulate emotions well. I’ve always been the opposite, hiding, because I knew I was different, but didn’t understand why.

The combination of the COVID-19 pandemic and my recent ADHD diagnosis have given me a different perspective. I now know why I hid. And now I want to engage, and be engaged, in the world.

If I can be a force for positive change, around people’s knowledge and opinion of ADHD, then I will.

If talking about Business Analysis (my day job), and sharing my ideas for optimising organisations helps anyone at all, then I will.

If I can show my photos and brighten someone’s day by allowing them to enjoy a sunset, or a flying bird, then I will.

And if anyone buys any of my photos, then I will be shocked!

So welcome to my little vanity project. I hope it can be something positive, for me, if for noone else in this new, odd world in which we now find ourselves living together.

,

,

,

Matt PalmerPrivate Key Redaction: UR DOIN IT RONG

Because posting private keys on the Internet is a bad idea, some people like to “redact” their private keys, so that it looks kinda-sorta like a private key, but it isn’t actually giving away anything secret. Unfortunately, due to the way that private keys are represented, it is easy to “redact” a key in such a way that it doesn’t actually redact anything at all. RSA private keys are particularly bad at this, but the problem can (potentially) apply to other keys as well.

I’ll show you a bit of “Inside Baseball” with key formats, and then demonstrate the practical implications. Finally, we’ll go through a practical worked example from an actual not-really-redacted key I recently stumbled across in my travels.

The Private Lives of Private Keys

Here is what a typical private key looks like, when you come across it:

-----BEGIN RSA PRIVATE KEY-----
MGICAQACEQCxjdTmecltJEz2PLMpS4BXAgMBAAECEDKtuwD17gpagnASq1zQTYEC
CQDVTYVsjjF7IQIJANUYZsIjRsR3AgkAkahDUXL0RSECCB78r2SnsJC9AghaOK3F
sKoELg==
-----END RSA PRIVATE KEY-----

Obviously, there’s some hidden meaning in there – computers don’t encrypt things by shouting “BEGIN RSA PRIVATE KEY!”, after all. What is between the BEGIN/END lines above is, in fact, a base64-encoded DER format ASN.1 structure representing a PKCS#1 private key.

In simple terms, it’s a list of numbers – very important numbers. The list of numbers is, in order:

  • A version number (0);
  • The “public modulus”, commonly referred to as “n”;
  • The “public exponent”, or “e” (which is almost always 65,537, for various unimportant reasons);
  • The “private exponent”, or “d”;
  • The two “private primes”, or “p” and “q”;
  • Two exponents, which are known as “dmp1” and “dmq1”; and
  • A coefficient, known as “iqmp”.

Why Is This a Problem?

The thing is, only three of those numbers are actually required in a private key. The rest, whilst useful to allow the RSA encryption and decryption to be more efficient, aren’t necessary. The three absolutely required values are e, p, and q.

Of the other numbers, most of them are at least about the same size as each of p and q. So of the total data in an RSA key, less than a quarter of the data is required. Let me show you with the above “toy” key, by breaking it down piece by piece1:

  • MGI – DER for “this is a sequence”
  • CAQ – version (0)
  • CxjdTmecltJEz2PLMpS4BXn
  • AgMBAAe
  • ECEDKtuwD17gpagnASq1zQTYd
  • ECCQDVTYVsjjF7IQp
  • IJANUYZsIjRsR3q
  • AgkAkahDUXL0RSdmp1
  • ECCB78r2SnsJC9dmq1
  • AghaOK3FsKoELg==iqmp

Remember that in order to reconstruct all of these values, all I need are e, p, and q – and e is pretty much always 65,537. So I could “redact” almost all of this key, and still give all the important, private bits of this key. Let me show you:

-----BEGIN RSA PRIVATE KEY-----
..............................................................EC
CQDVTYVsjjF7IQIJANUYZsIjRsR3....................................
........
-----END RSA PRIVATE KEY-----

Now, I doubt that anyone is going to redact a key precisely like this… but then again, this isn’t a “typical” RSA key. They usually look a lot more like this:

-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAu6Inch7+mWtKn+leB9uCG3MaJIxRyvC/5KTz2fR+h+GOhqj4
SZJobiVB4FrE5FgC7AnlH6qeRi9MI0s6dt5UWZ5oNIeWSaOOeNO+EJDUkSVf67wj
SNGXlSjGAkPZ0nRJiDjhuPvQmdW53hOaBLk5udxPEQbenpXAzbLJ7wH5ouLQ3nQw
HwpwDNQhF6zRO8WoscpDVThOAM+s4PS7EiK8ZR4hu2toon8Ynadlm95V45wR0VlW
zywgbkZCKa1IMrDCscB6CglQ10M3Xzya3iTzDtQxYMVqhDrA7uBYRxA0y1sER+Rb
yhEh03xz3AWemJVLCQuU06r+FABXJuY/QuAVvQIDAQABAoIBAFqwWVhzWqNUlFEO
PoCVvCEAVRZtK+tmyZj9kU87ORz8DCNR8A+/T/JM17ZUqO2lDGSBs9jGYpGRsr8s
USm69BIM2ljpX95fyzDjRu5C0jsFUYNi/7rmctmJR4s4uENcKV5J/++k5oI0Jw4L
c1ntHNWUgjK8m0UTJIlHbQq0bbAoFEcfdZxd3W+SzRG3jND3gifqKxBG04YDwloy
tu+bPV2jEih6p8tykew5OJwtJ3XsSZnqJMwcvDciVbwYNiJ6pUvGq6Z9kumOavm9
XU26m4cWipuK0URWbHWQA7SjbktqEpxsFrn5bYhJ9qXgLUh/I1+WhB2GEf3hQF5A
pDTN4oECgYEA7Kp6lE7ugFBDC09sKAhoQWrVSiFpZG4Z1gsL9z5YmZU/vZf0Su0n
9J2/k5B1GghvSwkTqpDZLXgNz8eIX0WCsS1xpzOuORSNvS1DWuzyATIG2cExuRiB
jYWIJUeCpa5p2PdlZmBrnD/hJ4oNk4oAVpf+HisfDSN7HBpN+TJfcAUCgYEAyvY7
Y4hQfHIdcfF3A9eeCGazIYbwVyfoGu70S/BZb2NoNEPymqsz7NOfwZQkL4O7R3Wl
Rm0vrWT8T5ykEUgT+2ruZVXYSQCKUOl18acbAy0eZ81wGBljZc9VWBrP1rHviVWd
OVDRZNjz6nd6ZMrJvxRa24TvxZbJMmO1cgSW1FkCgYAoWBd1WM9HiGclcnCZknVT
UYbykCeLO0mkN1Xe2/32kH7BLzox26PIC2wxF5seyPlP7Ugw92hOW/zewsD4nLze
v0R0oFa+3EYdTa4BvgqzMXgBfvGfABJ1saG32SzoWYcpuWLLxPwTMsCLIPmXgRr1
qAtl0SwF7Vp7O/C23mNukQKBgB89DOEB7xloWv3Zo27U9f7nB7UmVsGjY8cZdkJl
6O4LB9PbjXCe3ywZWmJqEbO6e83A3sJbNdZjT65VNq9uP50X1T+FmfeKfL99X2jl
RnQTsrVZWmJrLfBSnBkmb0zlMDAcHEnhFYmHFuvEnfL7f1fIoz9cU6c+0RLPY/L7
n9dpAoGAXih17mcmtnV+Ce+lBWzGWw9P4kVDSIxzGxd8gprrGKLa3Q9VuOrLdt58
++UzNUaBN6VYAe4jgxGfZfh+IaSlMouwOjDgE/qzgY8QsjBubzmABR/KWCYiRqkj
qpWCgo1FC1Gn94gh/+dW2Q8+NjYtXWNqQcjRP4AKTBnPktEvdMA=
-----END RSA PRIVATE KEY-----

People typically redact keys by deleting whole lines, and usually replacing them with [...] and the like. But only about 345 of those 1588 characters (excluding the header and footer) are required to construct the entire key. You can redact about 4/5ths of that giant blob of stuff, and your private parts (or at least, those of your key) are still left uncomfortably exposed.

But Wait! There’s More!

Remember how I said that everything in the key other than e, p, and q could be derived from those three numbers? Let’s talk about one of those numbers: n.

This is known as the “public modulus” (because, along with e, it is also present in the public key). It is very easy to calculate: n = p * q. It is also very early in the key (the second number, in fact).

Since n = p * q, it follows that q = n / p. Thus, as long as the key is intact up to p, you can derive q by simple division.

Real World Redaction

At this point, I’d like to introduce an acquaintance of mine: Mr. Johan Finn. He is the proud owner of the GitHub repo johanfinn/scripts. For a while, his repo contained a script that contained a poorly-redacted private key. He since deleted it, by making a new commit, but of course because git never really deletes anything, it’s still available.

Of course, Mr. Finn may delete the repo, or force-push a new history without that commit, so here is the redacted private key, with a bit of the surrounding shell script, for our illustrative pleasure:

#Add private key to .ssh folder
cd /home/johan/.ssh/
echo  "-----BEGIN RSA PRIVATE KEY-----
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
ÄÄÄÄÄÄÄÄÄÄÄÄÄÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::.::
:::::::::::::::::::::::::::.::::::::::::::::::::::::::::::::::::
LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLlL
ÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖÖ
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
ÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅÅ
YYYYYYYYYYYYYYYYYYYYYyYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
-----END RSA PRIVATE KEY-----" >> id_rsa

Now, if you try to reconstruct this key by removing the “obvious” garbage lines (the ones that are all repeated characters, some of which aren’t even valid base64 characters), it still isn’t a key – at least, openssl pkey doesn’t want anything to do with it. The key is very much still in there, though, as we shall soon see.

Using a gem I wrote and a quick bit of Ruby, we can extract a complete private key. The irb session looks something like this:

>> require "derparse"
>> b64 = <<EOF
MIIJKgIBAAKCAgEAxEVih1JGb8gu/Fm4AZh+ZwJw/pjzzliWrg4mICFt1g7SmIE2
TCQMKABdwd11wOFKCPc/UzRH/fHuQcvWrpbOSdqev/zKff9iedKw/YygkMeIRaXB
fYELqvUAOJ8PPfDm70st9GJRhjGgo5+L3cJB2gfgeiDNHzaFvapRSU0oMGQX+kI9
ezsjDAn+0Pp+r3h/u1QpLSH4moRFGF4omNydI+3iTGB98/EzuNhRBHRNq4oBV5SG
Pq/A1bem2ninnoEaQ+OPESxYzDz3Jy9jV0W/6LvtJ844m+XX69H5fqq5dy55z6DW
sGKn78ULPVZPsYH5Y7C+CM6GAn4nYCpau0t52sqsY5epXdeYx4Dc+Wm0CjXrUDEe
Egl4loPKDxJkQqQ/MQiz6Le/UK9vEmnWn1TRXK3ekzNV4NgDfJANBQobOpwt8WVB
rbsC0ON7n680RQnl7PltK9P1AQW5vHsahkoixk/BhcwhkrkZGyDIl9g8Q/Euyoq3
eivKPLz7/rhDE7C1BzFy7v8AjC3w7i9QeHcWOZFAXo5hiDasIAkljDOsdfD4tP5/
wSO6E6pjL3kJ+RH2FCHd7ciQb+IcuXbku64ln8gab4p8jLa/mcMI+V3eWYnZ82Yu
axsa85hAe4wb60cp/rCJo7ihhDTTvGooqtTisOv2nSvCYpcW9qbL6cGjAXECAwEA
AQKCAgEAjz6wnWDP5Y9ts2FrqUZ5ooamnzpUXlpLhrbu3m5ncl4ZF5LfH+QDN0Kl
KvONmHsUhJynC/vROybSJBU4Fu4bms1DJY3C39h/L7g00qhLG7901pgWMpn3QQtU
4P49qpBii20MGhuTsmQQALtV4kB/vTgYfinoawpo67cdYmk8lqzGzzB/HKxZdNTq
s+zOfxRr7PWMo9LyVRuKLjGyYXZJ/coFaobWBi8Y96Rw5NZZRYQQXLIalC/Dhndm
AHckpstEtx2i8f6yxEUOgPvV/gD7Akn92RpqOGW0g/kYpXjGqZQy9PVHGy61sInY
HSkcOspIkJiS6WyJY9JcvJPM6ns4b84GE9qoUlWVF3RWJk1dqYCw5hz4U8LFyxsF
R6WhYiImvjxBLpab55rSqbGkzjI2z+ucDZyl1gqIv9U6qceVsgRyuqdfVN4deU22
LzO5IEDhnGdFqg9KQY7u8zm686Ejs64T1sh0y4GOmGsSg+P6nsqkdlXH8C+Cf03F
lqPFg8WQC7ojl/S8dPmkT5tcJh3BPwIWuvbtVjFOGQc8x0lb+NwK8h2Nsn6LNazS
0H90adh/IyYX4sBMokrpxAi+gMAWiyJHIHLeH2itNKtAQd3qQowbrWNswJSgJzsT
JuJ7uqRKAFkE6nCeAkuj/6KHHMPsfCAffVdyGaWqhoxmPOrnVgECggEBAOrCCwiC
XxwUgjOfOKx68siFJLfHf4vPo42LZOkAQq5aUmcWHbJVXmoxLYSczyAROopY0wd6
Dx8rqnpO7OtZsdJMeBSHbMVKoBZ77hiCQlrljcj12moFaEAButLCdZFsZW4zF/sx
kWIAaPH9vc4MvHHyvyNoB3yQRdevu57X7xGf9UxWuPil/jvdbt9toaraUT6rUBWU
GYPNKaLFsQzKsFWAzp5RGpASkhuiBJ0Qx3cfLyirjrKqTipe3o3gh/5RSHQ6VAhz
gdUG7WszNWk8FDCL6RTWzPOrbUyJo/wz1kblsL3vhV7ldEKFHeEjsDGroW2VUFlS
asAHNvM4/uYcOSECggEBANYH0427qZtLVuL97htXW9kCAT75xbMwgRskAH4nJDlZ
IggDErmzBhtrHgR+9X09iL47jr7dUcrVNPHzK/WXALFSKzXhkG/yAgmt3r14WgJ6
5y7010LlPFrzaNEyO/S4ISuBLt4cinjJsrFpoo0WI8jXeM5ddG6ncxdurKXMymY7
EOF
>> b64 += <<EOF
gff0GJCOMZ65pMSy3A3cSAtjlKnb4fWzuHD5CFbusN4WhCT/tNxGNSpzvxd8GIDs
nY7exs9L230oCCpedVgcbayHCbkChEfoPzL1e1jXjgCwCTgt8GjeEFqc1gXNEaUn
O8AJ4VlR8fRszHm6yR0ZUBdY7UJddxQiYOzt0S1RLlECggEAbdcs4mZdqf3OjejJ
06oTPs9NRtAJVZlppSi7pmmAyaNpOuKWMoLPElDAQ3Q7VX26LlExLCZoPOVpdqDH
KbdmBEfTR4e11Pn9vYdu9/i6o10U4hpmf4TYKlqk10g1Sj21l8JATj/7Diey8scO
sAI1iftSg3aBSj8W7rxCxSezrENzuqw5D95a/he1cMUTB6XuravqZK5O4eR0vrxR
AvMzXk5OXrUEALUvt84u6m6XZZ0pq5XZxq74s8p/x1JvTwcpJ3jDKNEixlHfdHEZ
ZIu/xpcwD5gRfVGQamdcWvzGHZYLBFO1y5kAtL8kI9tW7WaouWVLmv99AyxdAaCB
Y5mBAQKCAQEAzU7AnorPzYndlOzkxRFtp6MGsvRBsvvqPLCyUFEXrHNV872O7tdO
GmsMZl+q+TJXw7O54FjJJvqSSS1sk68AGRirHop7VQce8U36BmI2ZX6j2SVAgIkI
9m3btCCt5rfiCatn2+Qg6HECmrCsHw6H0RbwaXS4RZUXD/k4X+sslBitOb7K+Y+N
Bacq6QxxjlIqQdKKPs4P2PNHEAey+kEJJGEQ7bTkNxCZ21kgi1Sc5L8U/IGy0BMC
PvJxssLdaWILyp3Ws8Q4RAoC5c0ZP0W2j+5NSbi3jsDFi0Y6/2GRdY1HAZX4twem
Q0NCedq1JNatP1gsb6bcnVHFDEGsj/35oQKCAQEAgmWMuSrojR/fjJzvke6Wvbox
FRnPk+6YRzuYhAP/YPxSRYyB5at++5Q1qr7QWn7NFozFIVFFT8CBU36ktWQ39MGm
cJ5SGyN9nAbbuWA6e+/u059R7QL+6f64xHRAGyLT3gOb1G0N6h7VqFT25q5Tq0rc
Lf/CvLKoudjv+sQ5GKBPT18+zxmwJ8YUWAsXUyrqoFWY/Tvo5yLxaC0W2gh3+Ppi
EDqe4RRJ3VKuKfZxHn5VLxgtBFN96Gy0+Htm5tiMKOZMYAkHiL+vrVZAX0hIEuRZ
EOF
>> der = b64.unpack("m").first
>> c = DerParse.new(der).first_node.first_child
>> version = c.value
=> 0
>> c = c.next_node
>> n = c.value
=> 80071596234464993385068908004931... # (etc)
>> c = c.next_node
>> e = c.value
=> 65537
>> c = c.next_node
>> d = c.value
=> 58438813486895877116761996105770... # (etc)
>> c = c.next_node
>> p = c.value
=> 29635449580247160226960937109864... # (etc)
>> c = c.next_node
>> q = c.value
=> 27018856595256414771163410576410... # (etc)

What I’ve done, in case you don’t speak Ruby, is take the two “chunks” of plausible-looking base64 data, chuck them together into a variable named b64, unbase64 it into a variable named der, pass that into a new DerParse instance, and then walk the DER value tree until I got all the values I need.

Interestingly, the q value actually traverses the “split” in the two chunks, which means that there’s always the possibility that there are lines missing from the key. However, since p and q are supposed to be prime, we can “sanity check” them to see if corruption is likely to have occurred:

>> require "openssl"
>> OpenSSL::BN.new(p).prime?
=> true
>> OpenSSL::BN.new(q).prime?
=> true

Excellent! The chances of a corrupted file producing valid-but-incorrect prime numbers isn’t huge, so we can be fairly confident that we’ve got the “real” p and q. Now, with the help of another one of my creations we can use e, p, and q to create a fully-operational battle key:

>> require "openssl/pkey/rsa"
>> k = OpenSSL::PKey::RSA.from_factors(p, q, e)
=> #<OpenSSL::PKey::RSA:0x0000559d5903cd38>
>> k.valid?
=> true
>> k.verify(OpenSSL::Digest::SHA256.new, k.sign(OpenSSL::Digest::SHA256.new, "bob"), "bob")
=> true

… and there you have it. One fairly redacted-looking private key brought back to life by maths and far too much free time.

Sorry Mr. Finn, I hope you’re not still using that key on anything Internet-facing.

What About Other Key Types?

EC keys are very different beasts, but they have much the same problems as RSA keys. A typical EC key contains both private and public data, and the public portion is twice the size – so only about 1/3 of the data in the key is private material. It is quite plausible that you can “redact” an EC key and leave all the actually private bits exposed.

What Do We Do About It?

In short: don’t ever try and redact real private keys. For documentation purposes, just put “KEY GOES HERE” in the appropriate spot, or something like that. Store your secrets somewhere that isn’t a public (or even private!) git repo.

Generating a “dummy” private key and sticking it in there isn’t a great idea, for different reasons: people have this odd habit of reusing “demo” keys in real life. There’s no need to encourage that sort of thing.


  1. Technically the pieces aren’t 100% aligned with the underlying DER, because of how base64 works. I felt it was easier to understand if I stuck to chopping up the base64, rather than decoding into DER and then chopping up the DER. 

,

Jonathan Adamczewskif32, u32, and const

Some time ago, I wrote “floats, bits, and constant expressions” about converting floating point number into its representative ones and zeros as a C++ constant expression – constructing the IEEE 754 representation without being able to examine the bits directly.

I’ve been playing around with Rust recently, and rewrote that conversion code as a bit of a learning exercise for myself, with a thoroughly contrived set of constraints: using integer and single-precision floating point math, at compile time, without unsafe blocks, while using as few unstable features as possible.

I’ve included the listing below, for your bemusement and/or head-shaking, and you can play with the code in the Rust Playground and rust.godbolt.org

// Jonathan Adamczewski 2020-05-12
//
// Constructing the bit-representation of an IEEE 754 single precision floating 
// point number, using integer and single-precision floating point math, at 
// compile time, in rust, without unsafe blocks, while using as few unstable 
// features as I can.
//
// or "What if this silly C++ thing https://brnz.org/hbr/?p=1518 but in Rust?"


// Q. Why? What is this good for?
// A. To the best of my knowledge, this code serves no useful purpose. 
//    But I did learn a thing or two while writing it :)


// This is needed to be able to perform floating point operations in a const 
// function:
#![feature(const_fn)]


// bits_transmute(): Returns the bits representing a floating point value, by
//                   way of std::mem::transmute()
//
// For completeness (and validation), and to make it clear the fundamentally 
// unnecessary nature of the exercise :D - here's a short, straightforward, 
// library-based version. But it needs the const_transmute flag and an unsafe 
// block.
#![feature(const_transmute)]
const fn bits_transmute(f: f32) -> u32 {
  unsafe { std::mem::transmute::<f32, u32>(f) }
}



// get_if_u32(predicate:bool, if_true: u32, if_false: u32):
//   Returns if_true if predicate is true, else if_false
//
// If and match are not able to be used in const functions (at least, not 
// without #![feature(const_if_match)] - so here's a branch-free select function
// for u32s
const fn get_if_u32(predicate: bool, if_true: u32, if_false: u32) -> u32 {
  let pred_mask = (-1 * (predicate as i32)) as u32;
  let true_val = if_true & pred_mask;
  let false_val = if_false & !pred_mask;
  true_val | false_val
}

// get_if_f32(predicate, if_true, if_false):
//   Returns if_true if predicate is true, else if_false
//
// A branch-free select function for f32s.
// 
// If either is_true or is_false is NaN or an infinity, the result will be NaN,
// which is not ideal. I don't know of a better way to implement this function
// within the arbitrary limitations of this silly little side quest.
const fn get_if_f32(predicate: bool, if_true: f32, if_false: f32) -> f32 {
  // can't convert bool to f32 - but can convert bool to i32 to f32
  let pred_sel = (predicate as i32) as f32;
  let pred_not_sel = ((!predicate) as i32) as f32;
  let true_val = if_true * pred_sel;
  let false_val = if_false * pred_not_sel;
  true_val + false_val
}


// bits(): Returns the bits representing a floating point value.
const fn bits(f: f32) -> u32 {
  // the result value, initialized to a NaN value that will otherwise not be
  // produced by this function.
  let mut r = 0xffff_ffff;

  // These floation point operations (and others) cause the following error:
  //     only int, `bool` and `char` operations are stable in const fn
  // hence #![feature(const_fn)] at the top of the file
  
  // Identify special cases
  let is_zero    = f == 0_f32;
  let is_inf     = f == f32::INFINITY;
  let is_neg_inf = f == f32::NEG_INFINITY;
  let is_nan     = f != f;

  // Writing this as !(is_zero || is_inf || ...) cause the following error:
  //     Loops and conditional expressions are not stable in const fn
  // so instead write this as type coversions, and bitwise operations
  //
  // "normalish" here means that f is a normal or subnormal value
  let is_normalish = 0 == ((is_zero as u32) | (is_inf as u32) | 
                        (is_neg_inf as u32) | (is_nan as u32));

  // set the result value for each of the special cases
  r = get_if_u32(is_zero,    0,           r); // if (iz_zero)    { r = 0; }
  r = get_if_u32(is_inf,     0x7f80_0000, r); // if (is_inf)     { r = 0x7f80_0000; }
  r = get_if_u32(is_neg_inf, 0xff80_0000, r); // if (is_neg_inf) { r = 0xff80_0000; }
  r = get_if_u32(is_nan,     0x7fc0_0000, r); // if (is_nan)     { r = 0x7fc0_0000; }
 
  // It was tempting at this point to try setting f to a "normalish" placeholder 
  // value so that special cases do not have to be handled in the code that 
  // follows, like so:
  // f = get_if_f32(is_normal, f, 1_f32);
  //
  // Unfortunately, get_if_f32() returns NaN if either input is NaN or infinite.
  // Instead of switching the value, we work around the non-normalish cases 
  // later.
  //
  // (This whole function is branch-free, so all of it is executed regardless of 
  // the input value)

  // extract the sign bit
  let sign_bit  = get_if_u32(f < 0_f32,  1, 0);

  // compute the absolute value of f
  let mut abs_f = get_if_f32(f < 0_f32, -f, f);

  
  // This part is a little complicated. The algorithm is functionally the same 
  // as the C++ version linked from the top of the file.
  // 
  // Because of the various contrived constraints on thie problem, we compute 
  // the exponent and significand, rather than extract the bits directly.
  //
  // The idea is this:
  // Every finite single precision float point number can be represented as a
  // series of (at most) 24 significant digits as a 128.149 fixed point number 
  // (128: 126 exponent values >= 0, plus one for the implicit leading 1, plus 
  // one more so that the decimal point falls on a power-of-two boundary :)
  // 149: 126 negative exponent values, plus 23 for the bits of precision in the 
  // significand.)
  //
  // If we are able to scale the number such that all of the precision bits fall 
  // in the upper-most 64 bits of that fixed-point representation (while 
  // tracking our effective manipulation of the exponent), we can then 
  // predictably and simply scale that computed value back to a range than can 
  // be converted safely to a u64, count the leading zeros to determine the 
  // exact exponent, and then shift the result into position for the final u32 
  // representation.
  
  // Start with the largest possible exponent - subsequent steps will reduce 
  // this number as appropriate
  let mut exponent: u32 = 254;
  {
    // Hex float literals are really nice. I miss them.

    // The threshold is 2^87 (think: 64+23 bits) to ensure that the number will 
    // be large enough that, when scaled down by 2^64, all the precision will 
    // fit nicely in a u64
    const THRESHOLD: f32 = 154742504910672534362390528_f32; // 0x1p87f == 2^87

    // The scaling factor is 2^41 (think: 64-23 bits) to ensure that a number 
    // between 2^87 and 2^64 will not overflow in a single scaling step.
    const SCALE_UP: f32 = 2199023255552_f32; // 0x1p41f == 2^41

    // Because loops are not available (no #![feature(const_loops)], and 'if' is
    // not available (no #![feature(const_if_match)]), perform repeated branch-
    // free conditional multiplication of abs_f.

    // use a macro, because why not :D It's the most compact, simplest option I 
    // could find.
    macro_rules! maybe_scale {
      () => {{
        // care is needed: if abs_f is above the threshold, multiplying by 2^41 
        // will cause it to overflow (INFINITY) which will cause get_if_f32() to
        // return NaN, which will destroy the value in abs_f. So compute a safe 
        // scaling factor for each iteration.
        //
        // Roughly equivalent to :
        // if (abs_f < THRESHOLD) {
        //   exponent -= 41;
        //   abs_f += SCALE_UP;
        // }
        let scale = get_if_f32(abs_f < THRESHOLD, SCALE_UP,      1_f32);    
        exponent  = get_if_u32(abs_f < THRESHOLD, exponent - 41, exponent); 
        abs_f     = get_if_f32(abs_f < THRESHOLD, abs_f * scale, abs_f);
      }}
    }
    // 41 bits per iteration means up to 246 bits shifted.
    // Even the smallest subnormal value will end up in the desired range.
    maybe_scale!();  maybe_scale!();  maybe_scale!();
    maybe_scale!();  maybe_scale!();  maybe_scale!();
  }

  // Now that we know that abs_f is in the desired range (2^87 <= abs_f < 2^128)
  // scale it down to be in the range (2^23 <= _ < 2^64), and convert without 
  // loss of precision to u64.
  const INV_2_64: f32 = 5.42101086242752217003726400434970855712890625e-20_f32; // 0x1p-64f == 2^64
  let a = (abs_f * INV_2_64) as u64;

  // Count the leading zeros.
  // (C++ doesn't provide a compile-time constant function for this. It's nice 
  // that rust does :)
  let mut lz = a.leading_zeros();

  // if the number isn't normalish, lz is meaningless: we stomp it with 
  // something that will not cause problems in the computation that follows - 
  // the result of which is meaningless, and will be ignored in the end for 
  // non-normalish values.
  lz = get_if_u32(!is_normalish, 0, lz); // if (!is_normalish) { lz = 0; }

  {
    // This step accounts for subnormal numbers, where there are more leading 
    // zeros than can be accounted for in a valid exponent value, and leading 
    // zeros that must remain in the final significand.
    //
    // If lz < exponent, reduce exponent to its final correct value - lz will be
    // used to remove all of the leading zeros.
    //
    // Otherwise, clamp exponent to zero, and adjust lz to ensure that the 
    // correct number of bits will remain (after multiplying by 2^41 six times - 
    // 2^246 - there are 7 leading zeros ahead of the original subnormal's
    // computed significand of 0.sss...)
    // 
    // The following is roughly equivalent to:
    // if (lz < exponent) {
    //   exponent = exponent - lz;
    // } else {
    //   exponent = 0;
    //   lz = 7;
    // }

    // we're about to mess with lz and exponent - compute and store the relative 
    // value of the two
    let lz_is_less_than_exponent = lz < exponent;

    lz       = get_if_u32(!lz_is_less_than_exponent, 7,             lz);
    exponent = get_if_u32( lz_is_less_than_exponent, exponent - lz, 0);
  }

  // compute the final significand.
  // + 1 shifts away a leading 1-bit for normal, and 0-bit for subnormal values
  // Shifts are done in u64 (that leading bit is shifted into the void), then
  // the resulting bits are shifted back to their final resting place.
  let significand = ((a << (lz + 1)) >> (64 - 23)) as u32;

  // combine the bits
  let computed_bits = (sign_bit << 31) | (exponent << 23) | significand;

  // return the normalish result, or the non-normalish result, as appopriate
  get_if_u32(is_normalish, computed_bits, r)
}


// Compile-time validation - able to be examined in rust.godbolt.org output
pub static BITS_BIGNUM: u32 = bits(std::f32::MAX);
pub static TBITS_BIGNUM: u32 = bits_transmute(std::f32::MAX);
pub static BITS_LOWER_THAN_MIN: u32 = bits(7.0064923217e-46_f32);
pub static TBITS_LOWER_THAN_MIN: u32 = bits_transmute(7.0064923217e-46_f32);
pub static BITS_ZERO: u32 = bits(0.0f32);
pub static TBITS_ZERO: u32 = bits_transmute(0.0f32);
pub static BITS_ONE: u32 = bits(1.0f32);
pub static TBITS_ONE: u32 = bits_transmute(1.0f32);
pub static BITS_NEG_ONE: u32 = bits(-1.0f32);
pub static TBITS_NEG_ONE: u32 = bits_transmute(-1.0f32);
pub static BITS_INF: u32 = bits(std::f32::INFINITY);
pub static TBITS_INF: u32 = bits_transmute(std::f32::INFINITY);
pub static BITS_NEG_INF: u32 = bits(std::f32::NEG_INFINITY);
pub static TBITS_NEG_INF: u32 = bits_transmute(std::f32::NEG_INFINITY);
pub static BITS_NAN: u32 = bits(std::f32::NAN);
pub static TBITS_NAN: u32 = bits_transmute(std::f32::NAN);
pub static BITS_COMPUTED_NAN: u32 = bits(std::f32::INFINITY/std::f32::INFINITY);
pub static TBITS_COMPUTED_NAN: u32 = bits_transmute(std::f32::INFINITY/std::f32::INFINITY);


// Run-time validation of many more values
fn main() {
  let end: usize = 0xffff_ffff;
  let count = 9_876_543; // number of values to test
  let step = end / count;
  for u in (0..=end).step_by(step) {
      let v = u as u32;
      
      // reference
      let f = unsafe { std::mem::transmute::<u32, f32>(v) };
      
      // compute
      let c = bits(f);

      // validation
      if c != v && 
         !(f.is_nan() && c == 0x7fc0_0000) && // nans
         !(v == 0x8000_0000 && c == 0) { // negative 0
          println!("{:x?} {:x?}", v, c); 
      }
  }
}

,

Chris NeugebauerReflecting on 10 years of not having to update WordPress

Over the weekend, the boredom of COVID-19 isolation motivated me to move my personal website from WordPress on a self-managed 10-year-old virtual private server to a generated static site on a static site hosting platform with a content delivery network.

This decision was overdue. WordPress never fit my brain particularly well, and it was definitely getting to a point where I wasn’t updating my website at all (my last post was two weeks before I moved from Hobart; I’ve been living in Petaluma for more than three years now).

Settling on which website framework wasn’t a terribly difficult choice (I chose Jekyll, everyone else seems to be using it), and I’ve had friends who’ve had success moving their blogs over. The difficulty I ended up facing was that the standard exporter that everyone to move from WordPress to Jekyll uses does not expect Debian’s package layout.

Backing up a bit: I made a choice, 10 years ago, to deploy WordPress on a machine that I ran myself, using the Debian system wordpress package, a simple aptitude install wordpress away. That decision was not particularly consequential then, but it chewed up 3 hours of my time on Saturday.

Why? The exporter plugin assumes that it will be able to find all of the standard WordPress files in the usual WordPress places, and when it didn’t find that, it broke in unexpected ways. And why couldn’t it find it?

Debian makes packaging choices that prioritise all the software on a system living side-by-side with minimal difficulty. It sets strict permissions. It separates application code from configuration from user data (which in the case of WordPress, includes plugins), in a way that is consistent between applications. This choice makes it easy for Debian admins to understand how to find bits of an application. It also minimises the chance of one PHP application from clobbering another.

10 years later, the install that I had set up was still working, having survived 3-4 Debian versions, and so 3-4 new WordPress versions. I don’t recall the last time I had to think about keeping my WordPress instance secure and updated. That’s quite a good run. I’ve had a working website despite not caring about keeping it updated for at least three years.

The same decisions that meant I spent 3 hours on Saturday doing a simple WordPress export saved me a bunch of time that I didn’t incrementally spend over the course a decade. Am I even? I have no idea.

Anyway, the least I can do is provide some help to people who might run into this same problem, so here’s a 5-step howto.

How to migrate a Debian WordPress site to Jekyll

Should you find the Jekyll exporter not working on your Debian WordPress install:

  1. Use the standard WordPress export to export an XML feel of your site.
  2. Spin up a new instance of WordPress (using WordPress.com, or on a new Virtual Private Server, whatever, really).
  3. Import the exported XML feed.
  4. Install the Jekyll exporter plugin.
  5. Follow the documentation and receive a Jekyll export of your site.

Basically, the plugin works with a stock WordPress install. If you don’t have one of those, it’s easy to move it over.

,

Gary PendergastInstall the COVIDSafe app

I can’t think of a more unequivocal title than that. 🙂

The Australian government doesn’t have a good track record of either launching publicly visible software projects, or respecting privacy, so I’ve naturally been sceptical of the contact tracing app since it was announced. The good news is, while it has some relatively minor problems, it appears to be a solid first version.

Privacy

While the source code is yet to be released, the Android version has already been decompiled, and public analysis is showing that it only collects necessary information, and only uploads contact information to the government servers when you press the button to upload (you should only press that button if you actually get COVID-19, and are asked to upload it by your doctor).

The legislation around the app is also clear that the data you upload can only be accessed by state health officials. Commonwealth departments have no access, neither do non-health departments (eg, law enforcement, intelligence).

Technical

It does what it’s supposed to do, and hasn’t been found to open you up to risks by installing it. There are a lot of people digging into it, so I would expect any significant issues to be found, reported, and fixed quite quickly.

Some parts of it are a bit rushed, and the way it scans for contacts could be more battery efficient (that should hopefully be fixed in the coming weeks when Google and Apple release updates that these contact tracing apps can use).

If it produces useful data, however, I’m willing to put up with some quirks. 🙂

Usefulness

I’m obviously not an epidemiologist, but those I’ve seen talk about it say that yes, the data this app produces will be useful for augmenting the existing contact tracing efforts. There were some concerns that it could produce a lot of junk data that wastes time, but I trust the expert contact tracing teams to filter and prioritise the data they get from it.

Install it!

The COVIDSafe site has links to the app in Apple’s App Store, as well as Google’s Play Store. Setting it up takes a few minutes, and then you’re done!

,

Andrew RuthvenInstall Fedora CoreOS using FAI

I've spent the last couple of days trying to deploy Fedora CoreOS to some physical hardware/bare metal for a colleague using the official PXE installer from Fedora CoreOS. It wasn't very pleasant, and just wouldn't work reliably.

Maybe my expectations were to high, in that I thought I could use Ignition to prepare more of the system for me, as my colleague has been able to bare metal installs correctly. I just tried to use Ignition as documented.

A few interesting aspects I encountered:

  1. The PXE installer for it has a 618MB initrd file. This takes quite a while to transfer via tftp!
  2. It can't build software RAID for the main install device (and the developers have no intention of adding this), and it seems very finicky to build other RAID sets for other partitions.
  3. And, well, I just kept having problems where the built systems would hang during boot for no obvious reason.
  4. The time to do an installation was incredibly long.
  5. The initrd image is really just running coreos-installer against the nominated device.

During the night I got feed up with that process and wrote a Fully Automatic Installer (FAI) profile that'd install CoreOS instead. I can now use setup-storage from FAI using it's standard disk_config files. This allows me to build complicated disk configurations with software RAID and LVM easily.

A big bonus is that a rebuild is a lot faster, timed from typing reboot to a fresh login prompt is 10 minutes - and this is on physical hardware so includes BIOS POST and RAID controller set up, twice each.

I thought this might be of interest to other people, so the FAI profile I developed for this is located here: https://github.com/catalyst-cloud/fai-profile-fedora-coreos

FAI was initially developed to deploy Debian systems, it has since been extended to be able to install a number of other operating systems, however I think this is a good example of how easy it is to deploy non-Debian derived operating systems using FAI without having to modify FAI itself.

,

Gary PendergastBebo, Betty, and Jaco

Wait, wasn’t WordPress 5.4 just released?

It absolutely was, and congratulations to everyone involved! Inspired by the fine work done to get another release out, I finally completed the last step of co-leading WordPress 5.0, 5.1, and 5.2 (Bebo, Betty, and Jaco, respectively).

My study now has a bit more jazz in it. 🙂

,

Robert CollinsStrength training from home

For the last year I’ve been incrementally moving away from lifting static weights and towards body weight based exercises, or callisthenics. I’ve been doing this for a number of reasons, including better avoidance of injury (if I collapse, the entire stack is dynamic, if a bar held above my head drops on me, most of the weight is just dead weight – ouch), accessibility during travel – most hotel gyms are very poor, and functional relevance – I literally never need to put 100 kg on my back, but I do climb stairs, for instance.

Covid-19 shutting down the gym where I train is a mild inconvenience for me as a result, because even though I don’t do it, I am able to do nearly all my workouts entirely from home. And I thought a post about this approach might be of interest to other folk newly separated from their training facilities.

I’ve gotten most of my information from a few different youtube channels:

There are many more channels out there, and I encourage you to go and look and read and find out what works for you. Those 5 are my greatest hits, if you will. I’ve bought the FitnessFAQs exercise programs to help me with my my training, and they are indeed very effective.

While you don’t need a gymnasium, you do need some equipment, particularly if you can’t go and use a local park. Exactly what you need will depend on what you choose to do – for instance, doing dips on the edge of a chair can avoid needing any equipment, but doing them with some portable parallel bars can be much easier. Similarly, doing pull ups on the edge of a door frame is doable, but doing them with a pull-up bar is much nicer on your fingers.

Depending on your existing strength you may not need bands, but I certainly did. Buying rings is optional – I love them, but they aren’t needed to have a good solid workout.

I bought parallettes for working on the planche.undefined Parallel bars for dips and rows.undefined A pull-up bar for pull-ups and chin-ups, though with the rings you can add flys, rows, face-pulls, unstable push-ups and more. The rings. And a set of 3 bands that combine for 7 different support amounts.undefinedundefined

In terms of routine, I do a upper/lower split, with 3 days on upper body, one day off, one day on lower, and the weekends off entirely. I was doing 2 days on lower body, but found I was over-training with Aikido later that same day.

On upper body days I’ll do (roughly) chin ups or pull ups, push ups, rows, dips, hollow body and arch body holds, handstands and some grip work. Today, as I write this on Sunday evening, 2 days after my last training day on Friday, I can still feel my lats and biceps from training Friday afternoon. Zero issue keeping the intensity up.

For lower body, I’ll do pistol squats, nordic drops, quad extensions, wall sits, single leg calf raises, bent leg calf raises. Again, zero issues hitting enough intensity to achieve growth / strength increases. The only issue at home is having a stable enough step to get a good heel drop for the calf raises.

If you haven’t done bodyweight training at all before, when starting, don’t assume it will be easy – even if you’re a gym junkie, our bodies are surprisingly heavy, and there’s a lot of resistance just moving them around.

Good luck, train well!

OpenSTEMOnline Teaching

The OpenSTEM® materials are ideally suited to online teaching. In these times of new challenges and requirements, there are a lot of technological possibilities. Schools and teachers are increasingly being asked to deliver material online to students. Our materials can assist with that process, especially for Humanities and Science subjects from Prep/Kindy/Foundation to Year 6. […]

The post Online Teaching first appeared on OpenSTEM Pty Ltd.

Brendan ScottCovid 19 Numbers – lag

Recording some thoughts about Covid 19 numbers.

Today’s figures

The Government says:

“As at 6.30am on 22 March 2020, there have been 1,098 confirmed cases of COVID-19 in Australia”.

The reference is https://www.health.gov.au/news/health-alerts/novel-coronavirus-2019-ncov-health-alert/coronavirus-covid-19-current-situation-and-case-numbers. However, that page is updated daily (ish), so don’t expect it to be the same if you check the reference.

Estimating Lag

If a person tests positive to the virus today, that means they were infected at some time in the past. So, what is the lag between infection and a positive test result?

Incubation Lag – about 5 days

When you are infected you don’t show symptoms immediately. Rather, there’s an incubation period before symptoms become apparent.  The time between being infected and developing symptoms varies from person to person, but most of the time a person shows symptoms after about 5 days (I recall seeing somewhere that 1 in a 1000 cases will develop symptoms after 14 days).

Presentation Lag – about 2 days

I think it’s fair to also assume that people are not presenting at testing immediately they become ill. It is probably taking them a couple of days from developing symptoms to actually get to the doctor – I read a story somewhere (have since lost the reference) about a young man who went to a party, then felt bad for days but didn’t go for a test until someone else from the party had returned a positive test.  Let’s assume there’s a mix of worried well and stoic types and call it 2 days from becoming symptomatic to seeking a test.

Referral Lag – about a day

Assuming that a GP is available straight away and recommends a test immediately, logistically there will still be most of a day taken up between deciding to see a doctor and having a test carried out.

Testing lag – about 2 days

The graph of infections “epi graph” today looks like this:

200322_new-and-cumulative-covid-19-cases-in-australia-by-notification-date_1

One thing you notice about the graph is that the new cases bars seem to increase for a couple of days, then decrease – so about 100 new cases in the last 24 hours, but almost 200 in the 24 hours before that. From the graph, the last 3 “dips” have been today (Sunday), last Thursday and last Sunday.  This seems to be happening every 3 to 4 days. I initially thought that the dips might mean fewer (or more) people presenting over weekends, but the period is inconsistent with that. I suspect, instead, that this actually means that testing is being batched.

That would mean that neither the peaks nor troughs is representative of infection surges/retreats, but is simply reflecting when tests are being processed. This seems to be a 4 day cycle, so, on average it seems that it would be about 2 days between having the test conducted and receiving a result. So a confirmed case count published today is actually showing confirmed cases as at about 2 days earlier.

Total lag

From the date someone is infected to the time that they receive a positive confirmation is about:

lag = time for symptoms to show+time to seek a test+referral time + time for the test to return a result

So, the published figures on confirmed infections are probably lagging actual infections in the community by about 10 days (5+2+1+2).

If there’s about a 10 day lag between infection and confirmation, then what a figure published today says is that about a week and a half ago there were about this many cases in the community.  So, the 22 March figure of 1098 infections is actually really a 12 March figure.

What the lag means for Physical (ie Social) Distancing

The main thing that the lag means is that if we were able to wave a magic wand today and stop all further infections, we would continue to record new infections for about 10 days (and the tail for longer). In practical terms, implementing physical distancing measures will not show any effect on new cases for about a week and a half. That’s because today there are infected people who are yet to be tested.

The silver lining to that is that the physical distancing measures that have been gaining prominence since 15 March should start to show up in the daily case numbers from the middle of the coming week, possibly offset by overseas entrants rushing to make the 20 March entry deadline.

Estimating Actual Infections as at Today

How many people are infected, but unconfirmed as at today? To estimate actual infections you’d need to have some idea of the rate at which infections are increasing. For example, if infections increased by 10% per day for 10 days, then you’d multiply the most recent figure by 1.1 raised to the power of 10 (ie about 2.5).  Unfortunately, the daily rate of increase (see table on the wiki page) has varied a fair bit (from 20% to 27%) over the most recent 10 days of data (that is, over the 10 days prior to 12 March, since the 22 March figures roughly correspond to 12 March infections) and there’s no guarantee that since that time the daily increase in infections will have remained stable, particularly in light of the implementation of physical distancing measures. At 23.5% per day, the factor is about 8.

There aren’t any reliable figures we can use to estimate the rate of infection during the current lag period (ie from 12 March to 22 March). This is because the vast majority of cases have not been from unexplained community transmission. Most of the cases are from people who have been overseas in the previous fortnight and they’re the cohort that has been most significantly impacted by recent physical distancing measures. From 15 March, they have been required to self isolate and from 20 March most of their entry into the country has stopped.  So I’d expect a surge in numbers up to about 30 March – ie reflecting infections in the cohort of people rushing to get into the country before the borders closed followed by a flattening. With the lag factor above, you’ll need to wait until 1 April or thereabouts to know for sure.

Note:

This post is just about accounting for the time lag between becoming infected and receiving a positive test result. It assumes, for example, that everyone who is infected seeks a test, and that everyone who is infected and seeks a test is, in fact, tested. As at today, neither of these things is true.

,

Ian BrownKubernetes Secrets Security

I was always of the belief that secrets within Kubernetes (k8s) are secure. How wrong I was! After a recent meetup featuring a Google security expert, I discovered that the secrets I have in our k8s cluster are no more secure than writing them down on paper and leaving it on a park bench. Google Cloud Platform is a fantastic offering, and yes, I am biased. I have used every other cloud platform from every major vendor over the last 10 odd years of my career.

,

Clinton Roylca2020 ReWatch 2020-02-02

As I was an organiser of the conference this year, I didn’t get to see many talks, fortunately many of the talks were recorded, so i get to watch the conference well after the fact.

Conference Opening

That white balance on the lectern slides is indeed bad, I really should get around to adding this as a suggestion on the logos documentation. (With some help, I put up all the lectern covers, it was therapeutic and rush free).

I actually think there was a lot of information in this introduction. Perhaps too much?

OpenZFS and Linux

A nice update on where zfs is these days.

Dev/Ops relationships, status: It’s Complicated

A bit of  a war story about production systems, leading to a moment of empathy.

Samba 2020: Why are we still in the 1980s for authentication?

There are a lot of old security standards that are showing there age, there are a lot of modern security standards, but which to choose?

Tyranny of the Clock

A very interesting problem solving adventure, with a few nuggets of interesting information about tools and techniques.

Configuration Is (riskier than?) Code

Because configuration files are parsed by a program, and the program changes how it runs depending on the contents of that configuration file, every program that parses configuration files is basically an interpreter, and thus every configuration file is basically a program. So, configuation is code, and we should be treating configuration like we do code, e.g. revision control, commenting, testing, review.

Easy Geo-Redundant Handover + Failover with MARS + systemd

Using a local process organiser to handle a cluster, interesting, not something I’d really promote. Not the best video cutting in this video, lots of time with the speaker pointing to his slides offscreen.

 

,

Robert Collins2019 in the rearview

2019 was a very busy year for us. I hadn’t realised how busy it was until I sat down to write this post. There’s also some moderately heavy stuff in here – if you have topics that trigger you, perhaps make sure you have spoons before reading.

We had all the usual stuff. Movies – my top two were Alita and Abominable though the Laundromat and Ford v Ferrari were both excellent and moving pieces. I introduced Cynthia to Teppanyaki and she fell in love with having egg roll thrown at her face hole.

When Cynthia started school we dropped gymnastics due to the time overload – we wanted some downtime for her to process after school, and with violin having started that year she was just looking so tired after a full day of school we felt it was best not to have anything on. Then last year we added in a specific learning tutor to help with the things that she approaches differently to the other kids in her class, giving 2 days a week of extra curricular activity after we moved swimming to the weekends.

At the end of last year she was finally chipper and with it most days after school, and she had been begging to get into more stuff, so we all got together and negotiated drama class and Aikido.

The drama school we picked, HSPA, is pretty amazing. Cynthia adored her first teacher there, and while upset at a change when they rearranged classes slightly, is again fully engaged and thrilled with her time there. Part of the class is putting on a full scale production – they did a version of the Happy Prince near the end of term 3 – and every student gets a part, with the ability for the older students to audition for more parts. On the other hand she tells me tonight that she wants to quit. So shrug, who knows :).

I last did martial arts when I took Aikido with sensei Darren Friend at Aikido Yoshinkai NSW back in Sydney, in the late 2000’s. And there was quite a bit less of me then. Cynthia had been begging to take a martial art for about 4 years, and we’d said that when she was old enough, we’d sign her up, so this year we both signed up for Aikido at the Rangiora Aikido Dojo. The Rangiora dojo is part of the NZ organisation Aikido Shinryukan which is part of the larger Aikikai style, which is quite different, yet the same, as the Yoshinkai Aikido that I had been learning. There have been quite a few moments where I have had to go back to something core – such as my stance – and unlearn it, to learn the Aikikai technique. Cynthia has found the group learning dynamic a bit challenging – she finds the explanations – needed when there are twenty kids of a range of ages and a range of experience – from new intakes each term through to ones that have been doing it for 5 or so years – get boring, and I can see her just switch off. Then she misses the actual new bit of information she didn’t have previously :(. Which then frustrates her. But she absolutely loves doing it, and she’s made a couple of friends there (everyone is positive and friendly, but there are some girls that like to play with her after the kids lesson). I have gotten over the body disconnect and awkwardness and things are starting to flow, I’m starting to be able to reason about things without just freezing in overload all the time, so that’s not bad after a year. However, the extra weight is making my forward rolls super super awkward. I can backward roll easily, with moderately good form; forward rolls though my upper body strength is far from what’s needed to support my weight through the start of the roll – my arm just collapses – so I’m in a sort of limbo – if I get the moment just right I can just start the contact on the shoulder; but if I get the moment slightly wrong, it hurts quite badly. And since I don’t want large scale injuries, doing the higher rolls is very unnerving for me. I suspect its 90% psychological, but am not sure how to get from where I am to having confidence in my technique, other than rinse-and-repeat. My hip isn’t affecting training much, and sensei Chris seems to genuinely like training with Cynthia and I, which is very nice: we feel welcomed and included in the community.

Speaking of my hip – earlier this year something ripped cartilage in my right hip – ended up having to have an MRI scan – and those machines sound exactly like a dot matrix printer – to diagnose it. Interestingly, having the MRI improved my symptoms, but we are sadly in hurry-up-and-wait mode. Before the MRI, I’d wake up at night with some soreness, and my right knee bent, foot on the bed, then sleepily let my leg collapse sideways to the right – and suddenly be awake in screaming agony as the joint opened up with every nerve at its disposal. When the MRI was done, they pumped the joint full of local anaesthetic for two purposes – one is to get a clean read on the joint, and the second is so that they can distinguish between referred surrounding pain, vs pain from the joint itself. It is to be expected with a joint issue that the local will make things feel better (duh), for up to a day or so while the local dissipates. The expression on the specialists face when I told him that I had had a permanent improvement trackable to the MRI date was priceless. Now, when I wake up with joint pain, and my leg sleepily falls back to the side, its only mildly uncomfortable, and I readjust without being brought to screaming awakeness. Similarly, early in Aikido training many activities would trigger pain, and now there’s only a couple of things that do. In another 12 or so months if the joint hasn’t fully healed, I’ll need to investigate options such as stem cells (which the specialist was negative about) or steroids (which he was more negative about) or surgery (which he was even more negative about). My theory about the improvement is that the cartilage that was ripped was sitting badly and the inflation for the MRI allowed it to settle back into the appropriate place (and perhaps start healing better). I’m told that reducing inflammation systematically is a good option. Turmeric time.

Sadly Cynthia has had some issues at school – she doesn’t fit the average mould and while wide spread bullying doesn’t seem to be a thing, there is enough of it, and she receives enough of it that its impacted her happiness more than a little – this blows up in school and at home as well. We’ve been trying a few things to improve this – helping her understand why folk behave badly, what to do in the moment (e.g. this video), but also that anything that goes beyond speech is assault and she needs to report that to us or teachers no matter what.

We’ve also had some remarkably awful interactions with another family at the school. We thought we had a friendly relationship, but I managed to trigger a complete meltdown of the relationship – not by doing anything objectively wrong, but because we had (unknown to me) different folkways, and some perfectly routine and normal behaviour turned out to be stressful and upsetting to them, and then they didn’t discuss it with us at all until it had brewed up in their heads into a big mess… and its still not resolved (and may not ever be: they are avoiding us both).

I weighed in at 110kg this morning. Jan the 4th 2019 I was 130.7kg. Feb 1 2018 I was 115.2kg. This year I peaked at 135.4kg, and got down to 108.7kg before Christmas food set in. That’s pretty happy making all things considered. Last year I was diagnosed with Coitus headaches and though I didn’t know it the medicine I was put on has a known side effect of weight gain. And it did – I had put it down to ongoing failure to manage my diet properly, but once my weight loss doctor gave me an alternative prescription for the headaches, I was able to start losing weight immediately. Sadly, though the weight gain through 2018 was effortless, losing the weight through 2019 was not. Doable, but not effortless. I saw a neurologist for the headaches when they recurred in 2019, and got a much more informative readout on them, how to treat and so on – basically the headaches can be thought of as an instability in the system, and the medicines goal is to stabilise things, and once stable for a decent period, we can attempt to remove the crutch. Often that’s successful, sometimes not, sometimes its successful on a second or third time. Sometimes you’re stuck with it forever. I’ve been eating a keto / LCHF diet – not super strict keto, though Jonie would like me to be on that, I don’t have the will power most of the time – there’s a local truck stop that sells killer hotdogs. And I simply adore them.

I started this year working for one of the largest companies on the planet – VMware. I left there in February and wrote a separate post about that. I followed that job with nearly the polar opposite – a startup working on a blockchain content distribution system. I wrote about that too. Changing jobs is hard in lots of ways – for instance I usually make friendships at my jobs, and those suffer some when you disappear to a new context – not everyone makes connections with you outside of the job context. Then there’s the somewhat non-rational emotional impact of not being in paid employment. The puritans have a lot to answer for. I’m there again, looking for work (and hey, if you’re going to be at Linux.conf.au (Gold Coast Australia January 13-17) I’ll be giving a presentation about some of the interesting things I got up to in the last job interregnum I had.

My feet have been giving me trouble for a couple of years now. My podiatrist is reasonably happy with my progress – and I can certainly walk further than I could – I even did some running earlier in the year, until I got shin splints. However, I seem to have hyper sensitive soles, so she can’t correct my pro-nation until we fix that, which at least for now means a 5 minute session where I touch my feet, someone else does, then something smooth then something rough – called “sensory massage”.

In 2017 and 2018 I injured myself at the gym, and in 2019 I wanted to avoid that, so I sought out ways to reduce injury. Moving away from machines was a big part of that; more focus on technique another part. But perhaps the largest part was moving from lifting dead weight to focusing on body weight exercises – callisthenics. This shifts from a dead weight to control when things go wrong, to an active weight, which can help deal with whatever has happened. So far at least, this has been pretty successful – although I’ve had minor issues – I managed to inflame the fatty pad the olecranon displaces when your elbow locks out – I’m nearly entirely transitioned to a weights-free program – hand stands, pistol squats, push ups, dead hangs and so on. My upper body strength needs to come along some before we can really go places though… and we’re probably going to max out the hamstring curl machine (at least for regular two-leg curls) before my core is strong enough to do a Nordic drop.

Lynne has been worried about injuring herself with weight lifting at the gym for some time now, but recently saw my physio – Ben Cameron at Pegasus PhysioSouth – who is excellent, and he suggested that she could have less chronic back pain if she took weights back up again. She’s recently told me that I’m allowed one ‘told you so’ about this, since she found herself in a spot where previously she would have put herself in a poor lifting position, but the weight training gave her a better option and she intuitively used it, avoiding pain. So that’s a good thing – complicated because of her bodies complicated history, but an excellent trainer and physio team are making progress.

Earlier this year she had a hell of a fright, with a regular eye checkup getting referred into a ‘you are going blind; maybe tomorrow, maybe within 10 years’ nightmare scenario. Fortunately a second opinion got a specialist who probably knows the same amount but was willing to communicate it with actual words… Lynne has a condition which diabetes (type I or II) can affect, and she has a vein that can alter state somewhat arbitrarily but will probably only degrade slowly, particularly if Lynne’s diet is managed as she has been doing.

Diet wise, Lynne also has been losing some weight but this is complicated by her chronic idiopathic pancreatitis. That’s code for ‘it keeps happening and we don’t know why’ pancreatitis. We’ve consulted a specialist in the North Island who comes highly recommended by Lynne’s GP, who said that rapid weight loss is a little known but possible cause of pancreatitis – and that fits the timelines involved. So Lynne needs to lose weight to manage the onset of type II diabetes. But not to fast, to avoid pancreatitis, which will hasten the onset of type II diabetes. Aiee. Slow but steady – she’s working with the same doctor I am for that, and a similar diet, though lower on the fats as she has no gall… bladder.

In April our kitchen waste pipe started chronically blocking, and investigation with a drain robot revealed a slump in the pipe. Ground penetrating radar reveal an anomaly under the garage… and this escalated. We’re going to have to move out of the house for a week while half the house’s carpets are lifted, grout is pumped into the foundations to tighten it all back up again – and hopefully they don’t over pump it – and then it all gets replaced. Oh, and it looks like the drive will be replaced again, to fix the slumped pipe permanently. It will be lovely when done but right now we’re facing a wall of disruption and argh.

Around September I think, we managed to have a gas poisoning scare – our gas hob was left on and triggered a fireball which fortunately only scared Lynne rather than flambéing her face. We did however not know how much exposure we’d had to the LPG, nor to partially combusted gas – which produces toxic CO as a by-product, so there was a trip into the hospital for observation with Cynthia, with Lynne opting out. Lynne and Cynthia had had plenty of the basic symptoms – headaches, dizziness and so on at the the time, but after waiting for 2 hours in the ER queue that had faded. Le sigh. The hospital, bless their cotton socks don’t have the necessary equipment to diagnose CO poisoning without a pretty invasive blood test, but still took Cynthia’s vitals using methods (manual observation and a infra-red reader) that are confounded by the carboxyhemoglobin that forms from the CO that has been inhaled. Pretty unimpressed – our GP was livid. (This is one recommended protocol). Oh, and our gas hob when we got checked out – as we were not sure if we had left it on, or it had been misbehaving, turned out to have never been safe, got decertified and the pipe cut at the regulator. So we’re cooking on a portable induction hob for now.

When we moved to Rangiora I was travelling a lot more, Christchurch itself had poorer air quality than Rangiora, and our financial base was a lot smaller. Now, Rangiora’s population has gone up nearly double (13k to 19k conservatively – and that’s ignoring the surrounds that use Rangiora as a base), we have more to work with, the air situation in Christchurch has improved massively, and even a busy years travel is less than I was doing before Cynthia came along. We’re looking at moving – we’re not sure where yet; maybe more country, maybe more city.

One lovely bright spot over the last few years has been reconnecting with friends from school, largely on Facebook – some of whom I had forgotten that I knew back at school – I had a little clique but was not very aware of the wider school population in hindsight (this was more than a little embarrassing to me, as I didn’t want to blurt out “who are you?!”) – and others whom I had not :). Some of these reconnections are just light touch person-X exists and cares somewhat – and that’s cool. One in particular has grown into a deeper friendship than we had back as schoolkids, and I am happy and grateful that that has happened.

Our cats are fat and happy. Well mostly. Baggy is fat and stressed and spraying his displeasure everywhere whenever the stress gets too much :(. Cynthia calls him Mr Widdlepants. The rest of the time he cuddles and purrs and is generally happy with life. Dibbler and Kitten-of-the-wild are relatively fine with everything.

Cynthia’s violin is coming along well. She did a small performance for her classroom (with her teacher) and wowed them. I’ve been inspired to start practising trumpet again. After 27 years of decay my skills are decidedly rusty, but they are coming along. Finding arrangements for violin + trumpet is a bit challenging, and my sight-reading-with-transposition struggles to cope, but we make do. Lynne is muttering about getting a clarinet or drum-kit and joining in.

So, 2019. Whew. I hope yours was less stressful and had as many or more bright points than ours. Onwards to 2020.

,

BlueHackersBlueHackers crowd-funding free psychology services at LCA and other conferences

BlueHackers has in the past arranged for a free counsellor/psychologist at several conferences (LCA, OSDC). Given the popularity and great reception of this service, we want to make this a regular thing and try to get this service available at every conference possible – well, at least Australian open source and related events.

Right now we’re trying to arrange for the service to be available at LCA2020 at the Gold Coast, we have excellent local psychologists already, and the LCA organisers are working on some of the logistical aspects.

Meanwhile, we need to get the funds organised. Fortunately this has never been a problem with BlueHackers, people know this is important stuff. We can make a real difference.

Unfortunately BlueHackers hasn’t yet completed its transition from OSDClub project to Linux Australia subcommittee, so this fundraiser is running in my personal name. Well, you know who I (Arjen) am, so I hope you’re ok all with that.

We have a little over a week until LCA2020 starts, let’s make this happen! Thanks. You can donate via MyCause.

The post BlueHackers crowd-funding free psychology services at LCA and other conferences first appeared on BlueHackers.org.

,

Robert CollinsA Cachecash retrospective

In June 2019 I started a new role as a software engineer at a startup called Cachecash. Today is probably the last day of payroll there, and as is my usual practice, I’m going to reflect back on my time there. Less commonly, I’m going to do so in public, as we’re about to open the code (yay), and its not a mega-corporation with everything shuttered up (also yay).

Framing

This is intended to be a blameless reflection on what has transpired. Blameless doesn’t mean inaccurate; but it means placing the focus on the process and system, not on the particular actor that happened to be wearing the hat at the time a particular event happened. Sometimes the system is defined by the actors, and in that case – well, I’ll let you draw your own conclusions if you encounter that case.

A retrospective that we can’t learn from is useless. Worse than useless, because it takes time to write and time to read and that time is lost to us forever. So if a thing is a particular way, it is going to get said. Not to be mean, but because false niceness will waste everyone’s time. Mine and my ex-colleagues whose time I respect. And yours, if you are still reading this.

What was Cachecash

Cachecash was a startup – still is in a very technical sense, corporation law being what it is. But it is still a couple of code bases – and a nascent open source project (which will hopefully continue) – built to operationalise and productise this research paper that the Cachecash founders wrote.

What it isn’t anymore is a company investing significant amounts of time and money in the form of engineering in making code, to make those code bases better.

Cachecash was also a team of people. That obviously changed over time, but at the time I write this it is:

  • Ghada
  • Justin
  • Kevin
  • Marcus
  • Petar
  • Robert
  • Scott

And we’re all pretty fantastic, if you ask me :).

Technical overview

The CAPNet paper that I linked above doesn’t describe a product. What it describes is a system that permits paying caches (think squid/varnish etc) for transmitting content to clients, while also detecting attempts by such caches to claim payment when they haven’t transmitted, or attempting to collude with a client to pretend to overtransmit and get paid that way. A classic incentives-aligned scheme.

Note that there is no blockchain involved at this layer.

The blockchain was added into this core system as a way to build a federated marketplace – the idea was that the blockchain provided a suitable substrate for negotiating the purchase and sale of contracts that would be audited using the CAPNet accounting system, the payments could be micropayments back onto the blockchain, and so on – we’d avoid the regular financial system, and we wouldn’t be building a fragile central system that would prevent other companies also participating.

Miners would mine coins, publishers would buy coins then place them in escrow as a promise to pay caches to deliver content to clients, and a client would deliver proof of delivery back to the cache which would then claim payment from the publisher.

Technical Challenges

There were a few things that turned up as significant issues. In no particular order:

The protocol

The protocol itself adds additional round trips to multiple peers – in its ‘normal’ configuration the client ends up running (web- for browers) GRPC connections to 5 endpoints (with all the normal windowing concerns, but potentially over QUIC), and then gets chunks of content in batches (concurrently) from 4 of the peers, runs a small crypto brute force operation on the combined result, and then moves onto the next group of content. This should be sounding suspiciously like TCP – it is basically a window management problem, and it has exactly the same performance management problems – fast start, maximum window size, how far to reduce it when problems are suffered. But accentuated: those 4 cache peers can all suffer their own independent noise problems, or be hostile. But also, they can also suffer correlated problems: they might all be in the same datacentre, or be all run by a hostile actor, or the client might be on a hostile WiFi link, or the client’s OS/browser might be hostile. Lets just say that there is a long, rich road for optimising this new protocol to make it fast, robust, reliable. Much as we have taken many years to make HTTP into QUIC, drawing upon techniques like forward error correction rather than retries – similar techniques will need to be applied to give this protocol similar performance characteristics. And evolving the protocol while maintaining the security properties is a complicated task, with three actors involved, who may collude in various ways.

An early performance analysis I did on the go code implementation showed that the brute forcing work was a bottleneck because while the time (once optimise) per second was entirely modest for any small amount of data, the delay added per window element acts as a brake on performance for high capacity low latency links. For a 1Gbps 25ms RTT link I estimated a need for 8 cores doing crypto brute forcing on the client.

JS

Cachecash is essentially implementing a new network protocol. There are some great hooks these days in browsers, and one can hook in and provide streams to things like video players to let them get one segment of video. However, for downloading an entire file – for instance, if one is downloading a full video, it is not so easy. This bug, open for 2 years now, is the standards based way to do it. Even so non-standards based way to do it involves buffering the entire content in memory, oh and reflecting everything through a static github service worker. (You of course host such a static page yourself, but then the whole idea of this federated distributed system breaks down a little).

Our initial JS implementation was getting under 512KBps with all-local servers – part of that was the bandwidth delay product issue mentioned above. Moving to getting chunks of content from each cache concurrently using futures improved that up to 512KBps, but thats still shocking for a system we want to be able to compete with the likes of Youtube, Cloudflare and Akamai.

One of the hot spots turned out to be calculating SHA-256 values – the CAPNet algorithm calculates thousands (it’s tunable, but 8k in the set I was analysing) of independent SHA’s per chunk of received data. This is a problem – in browser SHA routines, even the recent native hosted ones – are slow per SHA. They are not slow per byte. Most folk want to make a small number of SHA calculations. Maybe thousands in total. Not tens of thousands per MB of data received….. So we wrote an implementation of the core crypto routines in Rust WASM, which took our performance locally up to 2MBps in Firefox and 6MBps in Chromium.

It is also possible we’d show up as crypto-JS at that point and be blacklisted as malware!

Blockchain

Having chosen to involve a block chain in the stack we had to deal with that complexity. We chose to take bitcoin’s good bits and run with those rather than either running a sidechain, trying to fit new transaction types into bitcoin itself, or trying to shoehorn our particular model into e.g. Ethereum. This turned out to be a fairly large amount of work : not the core chain itself – cloning the parts of bitcoin that we wanted was very quick. But then layering on the changes that we needed, to start dealing with escrows and negotiating parameters between components and so forth. And some of the operational challenges below turned up here as well even just in developer test setups (in particular endpoint discovery).

Operational Challenges

The operational model was pretty interesting. The basic idea was that eventually there would be this big distributed system, a bit-coin like set of miners etc, and we’d be one actor in that ecosystem running some subset of the components, but that until then we’d be running:

  • A centralised ledger
  • Centralised random number generation for the micropayment system
  • Centralised deployment and operations for the cache fleet
  • Software update / vetting for the publisher fleet
  • Software update / publishing for the JS library
  • Some number of seed caches
  • Demo publishers to show things worked
  • Metrics, traces, chain explorer, centralised logging

We had most of this live and running in some fashion for most of the time I was there – we evolved it and improved it a number of times as we iterated on things. Where appropriate we chose open source components like Jaeger, Prometheus and Elasticsearch. We also added policy layers on top of them to provide rate limiting and anti-spoofing facilities. We deployed stuff in AWS, with EKS, and there were glitches and things to workaround but generally only a tiny amount of time went into that part of it. I think I spent a day on actual operations a month, or thereabouts.

Other parties were then expected to bring along additional caches to expand the network, additional publishers to expand the content accessible via the network, and clients to use the network.

Ensuring a process run by a third party is network reachable by a browser over HTTPS is a surprisingly non-simple problem. We partly simplified it by mandating that they run a docker container that we supplied, but there’s still the chance that they are running behind a firewall with asymmetric ingress. And after that we still need a domain name for their endpoint. You can give every cache a CNAME in a dedicated subdomain – say using their public key as the subdomain, so that only that cache can issue requests to update their endpoint information in DNS. It is all solvable, but doing it so that the amount of customer interaction and handholding is reduced to the bare minimum is important: a user with a fleet of 1000 machines doesn’t want to talk to us 1000 times, and we don’t want to talk to them either. But this was a another bit of this-isn’t-really-distributed-is-it grit in the distributed-ointment.

Adoption Challenges

ISPs with large fleets of machines are in principle happy to sell capacity on them in return for money – yay. But we have no revenue stream at the moment, so they aren’t really incentivised to put effort in, it becomes a matter of principle, not a fiscal “this is 10x better for my business” imperative. And right now, its 10x slower than HTTP. Or more.

Content owners with large amounts of content being delivered without a CDN would like a radically cheaper CDN. Except – we’re not actually radically cheaper on a cost structure basis. Current CDN’s are expensive for their expensive 2nd and third generation products because no-one offers what they offer – seamless in-request edge computing. But that ISP that is contributing a cache to the fleet is going to want the cache paid for, and thats the same cost structure as existing CDNs – who often have a free entry tier. We might have been able to make our network cheaper eventually, but I’m just not sure about the radically cheaper bit.

Content owners who would like a CDN marketplace where the CDN caches are competing with each other – driving costs down – rather than than the CDN operators competing – would absolutely love us. But I rather suspect that those owners want more sophisticated offerings. To be clear, I wasn’t on the customer development team, and didn’t get much in the way of customer development briefings. But things like edge computing workers, where completely custom code can run in the CDN network, adjacent to ones user, are much more powerful offerings than simple static content shipping offerings, and offered by all major CDN’s. These are trusted services – the CAPNet paper doesn’t solve the problem of running edge code and providing proof that it was run. Enarx might go some, or even a long way way to running such code in an untrusted context, but providing a proof that it was run – so that running it can become a mining or mining-like operation is a whole other question. Without such an answer, an edge computing network starts to depend on trusting the caches behaviour a lot more all over again – the network has no proof of execution to depend on.

Rapid adjustment – load spikes – is another possible use case, but the use of the blockchain to negotiate escrows actually seemed to work against our ability to offer that. Akami define load spike in a time frame faster than many block chains can decide that a transaction has actually been accepted. Offchain transactions are of course a known thing in the block chain space but again that becomes additional engineering.

Our use of a new network protocol – for all that it was layered on standard web technology – made it harder for potential content owners to adopt our technology. Rather than “we have 200 local proxies that will deliver content to your users, just generate a url of the form X.Y.Z”, our solution is “we do not trust the 200 local proxies that we have, so you need to run complicated JS in your browser/phone app etc” to verify that the proxies are actually doing their job. This is better in some ways – precisely because we don’t trust those proxies, but it also increases both the runtime cost of using the service, the integration cost adopting the service, and complexity of debugging issues receiving content via the service.

What did we learn?

It is said that “A startup is an organization formed to search for a repeatable and scalable business model.” What did we uncover in our search? What can we take away going forward?

In principle we have a classic two sided market – people with excess capacity close to users want to sell it, and people with excess demand for their content want to buy delivery capacity.

The baseline market is saturated. The market as a whole is on its third or perhaps fourth (depending on how you define things) major iteration of functionality.

Content delivery purchasers are ok with trusting their suppliers : any supply chain fraud happening in this space at the moment is so small no-one is talking about it that I heard about.

Some of the things we were doing don’t seem to have been important to the customers we talked to – I don’t have a great read on this, but in particular, the blockchain aspect seems to have been more important to our long term vision than to the 2-sided market place that we perceived. It would be fascinating to me to validate that somehow – would cache capacity suppliers be willing to trust us enough to sell capacity to us with just the auditing mechanism, without the blockchain? Would content providers be happy buying credit from us rather than from a neutral exchange?

What did I learn?

I think in hindsight my startup muscles were atrophied – it had been some years since Canonical and it took a few months to start really thinking lean-startup again on a personal basis. That’s ok, because I was hired to build systems. But its not great, because I can do better. So number one: think lean-startup and really step up to help with learning and validation.

I levelled up my Go lang skills. That was really nice – Kevin has deep knowledge there, and though I’ve written Go before I didn’t have a good appreciation for style or aesthetics, or why. I do now. Where before I’d say ‘I’m happy to dive in but its not a language I feel I really know’, I am now happy to say that I know Go. More to learn – there always is – but in a good place.

I did a similar thing to my JS skills, but not to the same degree. Having climbed fairly deeply into the JS client – which is written in Typescript, converted its bundling system to webpack to work better with Rust-WASM, and so on. Its still not my go-to place, but I’m much more comfortable there now.

And of course playing with Rust-WASM was pure delight. Markus and I are both Rust afficionados, and having a genuine reason to write some Rust code for work was just delightful. Finding this bug was just a bonus :).

It was also really really nice being back in a truely individual contributor role for a while. I really enjoyed being able to just fix bugs and get on with things while I got my bearings. I’ve ended up doing a bit more leadership – refining of requirements, translating between idea-and-specification and the like recently, but still about 80% of time has been able to be sit-down-and-code, and that really is a pleasant holiday.

What am I going to change?

I’m certainly going to get a new job :). If you’re hiring, hit me up. (If you don’t have my details already, linkedin is probably best).

I’m think there the core thing I need to do is more alignment of the day to day work I’m doing with needs of customer development : I don’t want to take on or take over the customer development role – that will often be done best in person with a customer for startups, and I’m happy remote – but the more I can connect what I’m trying to achieve with what will get the customers to pay us, the more successful any business I’m working in will be. This may be a case for non-vanity metrics, or talking more with the customer-development team, or – well, I don’t know exactly what it will look like until I see the context I end up in, but I think more connection will be important.

And I think the second major thing is to find a better balance between individual contribution and leadership. I love individual contribution, it is perhaps the least stressful and most Zen place to be. But it is also the least effective unless the project has exactly one team member. My most impactful and successful roles have been leadership roles, but the pure leadership role with no individual contribution slowly killed me inside. Pure individual contribution has been like I imagine crack to be, and perhaps just as toxic in the long term.

,

Robert CollinsRust and distributions

Daniel wrote a lovely blog post about Rust’s ability to be included in distributions, both as a language that you can get via the distribution, and as the language that components of the distribution are being written in.

I think this is a great goal to raise and I have just a few thoughts and quibbles. First I want to acknowledge and agree with him on the Rust community, its so very nice, and he is doing a great thing as rustup lead; I wish I had more time to put in, I have more things I want to contribute to rustup. I’ll try to get back to the meetings soon.

On trust

I completely agree about the need for the crates index improvement : without those we cannot have a mirror network, and thats a significant issue for offline users and slow-region users.

On curlsh though

It isn’t the worst possible thing, for all that its “untrusted bootstrapping”, the actual thing downloaded is https secured etc, and so is the rustup binary itself. Put another way, I think the horror is more perceptual than analyzed risk. Someone that trusts Verisign etc enough to download the Debian installer enough over it, has exactly the same risk as someone trusting Verisign enough to download rustup at that point in time.

Cross signing curlsh that with per-distro keys or something seems pretty ridiculous to me, since the root of trust is still that first download; unless you’re wandering up to someone who has bootstrapped their compiler by hand (to avoid reflections-on-trust attacks), to get an installer, to build a system, to then do reproducible builds, to check that other systems are actually safe… aieeee.

I think its easier to package the curl|sh shell script in Debian itself perhaps? apt install get-rustup; then if / when rustup becomes packaged the user instructions don’t change but the root of trust would, as get-rustup would be updated to not download rustup, but to trigger a different package install, and so forth.

I don’t think its desirable though, to have distribution forks of the contents that rustup manages – Debian+Redhat+Suse+… builds of nightly rust with all the things failing or not, and so on – I don’t see who that would help. And if we don’t have that then the root of trust would still not be shifted under the GPG keychain – it would still be the HTTPS infrastructure for downloading rust toolchains + the integrity of the rustup toolchain builds themselves. Making rustup, which currently shares that trust, have a different trust root, seems pointless.

On duplication of dependencies

I think Debian needs to become more inclusive here, not Rustup. Debian has spent; pauses, counts, yes, DECADES, rejecting multiple entire ecosystems because of a prejuidiced view about what the Right Way to manage dependencies is. And they are not right in a universal sense. They were right in an engineering sense: given constraints (builds are expensive, bandwidth is expensive, disk is expensive), they are right. But those are not universal constraints, and seeking to impose those constraints on Java and Node – its been an unmitigated disaster. It hasn’t made those upstreams better, or more secure, or systematically fixed problems for users. I have another post on this so rather than repeating I’m going to stop here :).

I think Rust has – like those languages – made the crucial, maintainer and engineering efficiency important choice to embrace enabling incremental change across libraries, with the consequence that dependencies don’t shift atomically, and sure, this is basically incompatible with Debian packaging world view which says that point and patch releases of libraries are not distinct packages, and thus the shared libs for these things all coexist in the same file on disk. Boom! Crash!

I assert that it is entirely possible to come up with a reasonable design for managing a respository of software that doesn’t make this conflation, would allow actual point and patch releases of exist as they are for the languages that have this characteristic, and be amenable to automation, auditing and reporting for security issues. E.g. Modernise Debian to cope with this fundamentally different language design decision… which would make Java and Node and Rust work so very much better.

Alternatively, if Debian doesn’t want to make it possible to natively support languages that have made this choice, Debian could:

  • ship static-but-for-system-libs builds
  • not include things written in rust
  • ask things written in rust to converge their dependencies again and again and again (and only update them when the transitive dependencies across the entire distro have converged)

I have a horrible suspicion about which Debian will choose to do :(. The blinkers / echo chamber are so very strong in that community.

For Windows

We got to parity with Linux for IO for non-McAfee users, but I guess there are a lot of them out there; we probably need to keep pushing on tweaking it until it work better for them too; perhaps autodetect McAfee and switch to minimal? I agree that making Windows users – like I am these days – feel tier one, would be nice :). Maybe a survey of user experience would be a good starting point.

Shared libraries

Perhaps generating versioned symbols automatically and building many versions of the crate and then munging them together? But I’d also like to point here again that the whole focus on shared libraries is a bit of a distribution blind spot, and looking at the vast amount of distribution of software occuring in app stores and their model, suggests different ways of dealing with these things. See also the fairly specific suggestion I make about the packaging system in Debian that is the root of the problem in my entirely humble view.

Bonus

John Goerzen posted an entirely different thing recently, but in it he discusses programs that don’t properly honour terminfo. Sadly I happen to know that large chunks of the Rust ecosystem assume that everything is ANSI these days, and it certainly sounds like, at least for John, that isn’t true. So thats another way in which Rust could be more inclusive – use these things that have been built, rather than being modern and new age and reinventing the 95% match.

,

Robert CollinsWant me to work with you?

Reach out to me – I’m currently looking for something interesting to do. https://www.linkedin.com/in/rbtcollins/ and https://twitter.com/rbtcollins are good ways to grab me if you don’t already have my details.

Should you reach out to me? Maybe :). First, a little retrospective.

Three years ago, I wrote the following when reflecting on what I wanted to be doing:

Priorities (roughly ordered most to least important):

  • Keep living in Rangiora (family)
  • Up to moderate travel requirements – 4 trips a year + LCA/PyCon
  • Significant autonomy (not at the expense of doing the right thing for the company, just I work best with the illusion of free will 🙂 )
  • Be doing something that matters
    • -> Being open source is one way to this, but not the only one
  • Something cutting edge would be awesome
    • -> Rust / Haskell / High performance requirements / scale / ….
  • Salary

How well did that work for me? Pretty good. I had a good satisfying job at VMware for 3 years, met some wonderful people, achieved some very cool things. And those priorities above were broadly achieved.
The one niggle that stands out was this – Did the things we were doing matter? Certainly there was no social impact – VMware isn’t a non-profit, being right at the core of capitalism as it is. There was direct connection and impact with the team, the staff we worked with and the users of the products… but it is just a bit hard to feel really connected through that though: VMware is a very large company and there are many layers between users and developers.

We were quite early adopters of Kubernetes, which allowed me to deepen my Go knowledge and experience some more fun with AWS scale operations. I had many interesting discussions about the relative strengths of Python Go and Rust and Java with colleagues there. (Hi Geoffrey).

Company culture is very important to me, and VMware has a fantastically supportive culture. One of the most supportive companies I’ve been in, bar none. It isn’t a truely remote-organised company though: rather its a bunch of offices that talk to each other, which I think is sad. True remote-first offers so much more engagement.

I enjoy building things to solve problems. I’ve either directly built, or shaped what is built, in all my most impactful and successful roles. Solving a problem once by hand is fine; solving it for years to come by creating a tool is far more powerful.

I seem to veer into toolmaking very often: giving other people the ability to solve their problems takes the power of a tool and multiplies it even further.

It should be no surprise then that I very much enjoy reading white papers like the original Dapper and Map-reduce ones, LinkedIn’s Kafka or for more recent fodder the Facebook Akkio paper. Excellent synthesis and toolmaking applied at industrial scale. I read those things and I want to be a part of the creation of those sorts of systems.

I was fortunate enough to take some time to go back to university part-time, which though logistically challenging is something I want to see through.

Thus I think my new roughly ordered (descending) list of priorities needs to be something like this:

  • Keep living in Rangiora (family)
  • Up to moderate travel requirements – 4 team-meeting trips a year + 2 conferences
  • Significant autonomy (not at the expense of doing the right thing for the company, just I work best with the illusion of free will 🙂 )
  • Be doing something that matters
    • Be working directly on a problem / system that has problems
  • Something cutting edge would be awesome
    • Rust / Haskell / High performance requirements / scale / ….
  • A generative (Westrum definition) + supportive company culture
  • Remote-first or at least very remote familiar environment
  • Support my part time study / self improvement initiative
  • Salary

,

Clinton RoyRestricted Sleep Regime

Since moving down to Melbourne my poor sleep has started up again. It’s really hard to say what the main factor driving this is. My doctor down here has put me onto a drug free way of trying to improve my sleep, and I think I kind of like it, while it’s no silver bullet, it is something I can go back to if I’m having trouble with my sleep, without having to get a prescription.

The basic idea is to maximise sleep efficiency. If you’re only getting n hours sleep a night, only spend n hours  a night in bed. This forces you to stay up and go to bed rather late for a few nights. Hopefully, being tired will help you sleep through the night in one large segment. Once you’ve successfully slept through the night a few times, relax your bed time by say fifteen minutes, and get used to that. Slowly over time, you increase the amount of sleep you’re getting, while keeping your efficiency high.

,

OpenSTEMElection Activity Bundle

With the upcoming federal election, many teachers want to do some related activities in class – and we have the materials ready for you! To make selecting suitable resources a bit easier, we have an Election Activity Bundle containing everything you need, available for just $9.90. Did you know that the secret ballot is an Australian […]

The post Election Activity Bundle first appeared on OpenSTEM Pty Ltd.