Max Humber's #code Posts

Updating Sankey

Wed, 04 Dec 2024 00:00:00 -0000

Just released Sankey 2.0, an open-source package for building Sankey diagrams in SwiftUI.

What?

A Sankey diagram visualizes flow: where stuff is coming from, where it’s going, and how much stuff is moving. Thick and thin bands represent the volume of flow. Sankey diagrams are perfect for showing money, user behavior, energy usage—basically anything involving inputs, outputs, and the in-between.

Why + What’s New?

I originally built Sankey 1.0 in May 2022 for a contract project (no other options existed at the time). Then I forgot about it—until a couple of weeks ago when I needed a Sankey diagram for a new app.

This new project requires offline rendering and Dark Mode support. So, while adding these features to Sankey, I also streamlined the API to make it dead simple to create beautiful charts like this:

With code that is a simple as this:

import Sankey
import SwiftUI

struct ContentView: View {
    let data = SankeyData(
        nodes: [
            SankeyNode("A", color: .blue),
            SankeyNode("B", color: .purple),
            SankeyNode("X", color: .red),
            SankeyNode("Y", color: .yellow),
            SankeyNode("Z", color: .green),
        ],
        links: [
            SankeyLink(5, from: "A", to: "X"),
            SankeyLink(7, from: "A", to: "Y"),
            SankeyLink(6, from: "A", to: "Z"),
            SankeyLink(2, from: "B", to: "X"),
            SankeyLink(9, from: "B", to: "Y"),
            SankeyLink(4, from: "B", to: "Z"),
        ]
    )

    var body: some View {
        SankeyDiagram(data)
            .nodeOpacity(0.9)
            .linkColorMode(.gradient)
            .padding(10)
            .frame(height: 350)
    }
}

#Preview {
    ContentView()
}

All “Old” Code is “Bad” Code

Revisiting old code is always a fun (and humbling) exercise. Because the code in 1.0 was bad! (If you’re not embarrassed by your past work, are you even learning?)

While the core feature of Sankey—connecting source nodes to target nodes— remains 90% of the code in 2.0 is new. Mostly, this is a consequence of replacing the Google Charts rendering engine with D3.js. But also because I removed a lot of the over-engineered “organization”, limited excessive configuration options, and fixed a lot of color handling mistakes.

Despite these major changes, I was able to maintain a good bit of backwards compatibility (the original Quickstart still works)!

Too Much Structure!

The Sankey package should be simple. It’s just a SankeyDiagram SwiftUI component. And a few other structs that help in its construction. In 1.0, I went overboard trying to organize everything for “future extensibility”, creating this convoluted mess:

--Package.swift
--Sources
----Sankey/
------Options/
--------Tooltip/
----------TextStyle/
------------SankeyOptions.Tooltip.TextStyle.swift
----------SankeyOptions.Tooltip.swift
--------Sankey/
----------SankeyOptions.Sankey.swift
----------Link/
------------Color/
--------------SankeyOptions.Sankey.Link.Color.swift
------------SankeyOptions.Sankey.Link.swift
------------ColorMode/
--------------SankeyOptions.Sankey.Link.ColorMode.swift
----------Node/
------------Label/
--------------SankeyOptions.Sankey.Node.Label.swift
------------ColorMode/
--------------SankeyOptions.Sankey.Node.ColorMode.swift
------------SankeyOptions.Sankey.Node.swift
--------SankeyOptions.swift
--------SankeyOptions+CustomStringConvertible.swift
--------SankeyOptions+init.swift
------Diagram/
--------SankeyDiagram.swift
--------SankeyDiagram+init.swift
------Link/
--------SankeyLink.swift
--------SankeyLink+ExpressibleByArrayLiteral.swift
--------SankeyLink+CustomStringConvertible.swift
------Node/
--------SankeyNode.swift

Now that we’re in “the future” I can say that all this “organization” wasn’t just unhelpful—it actively hindered my ability add to and update the package! For 2.0, I streamlined as much as possible:

--Package.swift
--Sources/
----Deprecated/
------SankeyDiagram+deprecated.swift
------SankeyLink+deprecated.swift
----Helpers/
------Color+.swift
------HexColor.swift
----Resources/
------d3.min.js
------d3-sankey.min.js
----SankeyData.swift
----SankeyDiagram.swift
----SankeyLink.swift
----SankeyNode.swift
----SankeyOptions.swift
----SankeyResources.swift

Easier to work with. But more importantly, easier to delete! Because, let’s be honest—future me will probably throw it all out in another two years!

“WWAD”

While the original “structure” slowed me down, exposing too many configuration options to users was an even bigger mistake. Look at this old init signature:

SankeyDiagram(
    _ data: [SankeyLink],
    nodeColors: [String]? = nil,
    nodeColorMode: SankeyOptions.Sankey.Node.ColorMode = .unique,
    nodeWidth: Double? = nil,
    nodePadding: Double? = nil,
    nodeLabelColor: String = "black",
    nodeLabelFontSize: Double = 24,
    nodeLabelFontName: String? = nil,
    nodeLabelBold: Bool = false,
    nodeLabelItalic: Bool = false,
    nodeLabelPadding: Double? = nil,
    nodeInteractivity: Bool = false,
    linkColors: [String]? = nil,
    linkColorMode: SankeyOptions.Sankey.Link.ColorMode? = nil,
    linkColorFill: String? = nil,
    linkColorFillOpacity: Double? = nil,
    linkColorStroke: String? = nil,
    linkColorStrokeWidth: Double = 0,
    tooltipValueLabel: String = "",
    tooltipTextColor: String = "black",
    tooltipTextFontSize: Double = 24,
    tooltipTextFontName: String? = nil,
    tooltipTextBold: Bool = false,
    tooltipTextItalic: Bool = false,
    layoutIterations: Int = 32
)

Can you tell I was just blindly trying to recreating someone else’s API?

Now when I build for SwiftUI I typically ask myself: “What Would Apple Do?” (WWAD). If this piece of code were native Apple component, what would it look and feel like? I must say, I think I nailed it in 2.0:

SankeyDiagram(data)
    .nodeAlignment(.justify)
    .nodeWidth(15)
    .nodePadding(20)
    .nodeDefaultColor(.gray)
    .nodeOpacity(0.8)
    .linkDefaultColor(.gray)
    .linkOpacity(0.7)
    .linkColorMode(nil)
    .labelPadding(8)
    .labelColor(.primary)
    .labelOpacity(0.9)
    .labelFontSize(14)
    .labelFontFamily("Times")

All customization options are now exposed as “Modifiers” on the SankeyDiagram object itself. This is enabled by hiding the SankeyOptions struct from the user and exposing modifiers that look like this:

// ...
public func nodeOpacity(_ value: Double) -> SankeyDiagram {
    var new = self
    new.options.nodeOpacity = value
    return new
}
// ...

I also replaced all externally facing hex codes with native SwiftUI colors. While implementing this was complex, I thought it was better to handle the complexity myself than burden users with it. Now Dark Mode and using a “color” like Color.primary will just work!

Conclusion

The code in Sankey 2.0 is easier to read, use, and built to be thrown away! While it’s not perfect—future me will probably laugh at it in the future—it works well today. And that’s all that matters. If you need to visualize flows in any of your apps I hope you give it a try!

The Case Against TCA

Wed, 27 Nov 2024 00:00:00 -0000

The Composable Architecture (TCA) is an iOS app development framework that promises “better state management”, “modular design”, and “testable side effects”. After doing some consulting with a startup looking to adopt the framework, I can confidently say: TCA creates more problems than it solves.

Pain.

TCA is packed with unnecessary complexity, redundant abstractions, and frustrating quirks. The startup I worked with ultimately abandoned the framework for the following reasons:

Steep Learning Curve: Requires mastering “reducers” and “stores” making onboarding difficult.
Excessive Boilerplate: Even simple features require excessive code, slowing development.
Performance Problems: Slow builds, indexing, and crashing Xcode previews waste time.
Compiler Issues: Reducers often break the compiler with hard-to-diagnose errors.
Maintenance Burden: Frequent updates break code and demand costly refactoring.
Hard to Remove: Hard to rip out once adopted.
Third-Party Risk: Maintained by essentially two people with no long-term guarantee.
Outpaced by SwiftUI: SwiftUI now handles most problems TCA was built to solve.

I contend that the only good idea in TCA is the DependencyClient pattern. But you don’t actually need TCA to implement the pattern, you can do it with just a few lines of swift (see my post on the Client Pattern).

Obnoxious Dependencies

Importing TCA brings in 16 additional dependencies, most of which are other Point-Free libraries. While these libraries might have the swift- prefix, only swift-collections and swift-syntax are actually from Apple:

swift-composable-architecture 1.16.1
  ├── combine-schedulers 1.0.2
  ├── swift-case-paths 1.5.5
  ├── swift-clocks 1.0.5
  ├── swift-collections 1.1.3
  ├── swift-concurrency-extras 1.2.0
  ├── swift-custom-dump 1.3.3
  ├── swift-dependencies 1.4.0
  ├── swift-identified-collections 1.1.0
  ├── swift-macro-testing 0.5.2
  ├── swift-navigation 2.2.2
  ├── swift-perception 1.3.5
  ├── swift-snapshot-testing 1.17.5
  ├── swift-syntax 600.0.0-prerelease-2024-09-04
  ├── SwiftDocCPlugin 1.4.3
  ├── SymbolKit 1.0.0
  └── xctest-dynamic-overlay 1.4.0

This level of bloat is overkill for what TCA claims to solve, and the naming convention (swift-navigation instead of point-free-navigation) feels super obnoxious and deliberately disingenuous.

Online Sentiment

Looking through reddit (and elsewhere online) it seems that my frustrations are widely shared:

Reddit: Have you used TCA in production?

“TCA is basically an extra layer of complexity in order to do what SwiftUI already does under the hood.”
“Watch the first three [TCA] videos on how to make a checklist… The solution is sooooo complicated.“
“[TCA] seems needlessly complex to jump through 5+ files just to figure out what’s happening on one simple screen.”

Reddit: I hate the Composable Architecture

“[TCA] adds unnecessary complexity and a central dependency.”
“Everything [in TCA] is so tightly connected, any change leads to changes everywhere else.“
“Working two years with that [TCA] shit was one of the reasons I quit the project.“

Rod Schmidt: Composable Architecture Experience

“TCA is a 3rd party framework. This means Apple doesn’t support it or care about it.“
“You have to constantly re-learn things as the [TCA] framework gets updated.”
“You can be much more productive with MVVM and get the same benefits [as TCA].”

Survey Results

Despite the criticism TCA is pretty popular. So who actually uses the framework? And why do they like it? To find out, I put up a survey in a few iOS/Swift subreddits and got 100 responses:

TCA Opinion	SwiftUI-first	UIKit-first
I dont like it and avoid using it	8	18
I like it a lot and prefer it for app architecture	5	19
Its acceptable but not my preferred choice	1	19
Not applicable (I havent used TCA)	3	27

Key insights:

83% of survey respondents started with UIKit, while only 17% started with SwiftUI.
47% of SwiftUI-first developers avoid TCA, compared to only 22% of UIKit-first developers.

TCA’s appeal seems rooted in its familiarity to UIKit-first developers, offering a structured, UIKit-like experience. For SwiftUI-first developers, TCA likely feels unnecessary and redundant, solving problems that SwiftUI already natively solves.

Vanilla

So what’s the alternative? Honestly, just vanilla SwiftUI! Frustrated with TCA, I decided to recreate the SpeechRecognition example to prove that the framework is unnecessary.

Check out my vanilla implementation. Most of the code is in a single file (just for ease of comparison) and works seamlessly with Xcode 16, Swift 6, and iOS 18. Here are the important benchmarks:

Metric	Vanilla	TCA
Dependencies	0	16
“Cold” Build Time (seconds)	1.1	32.4
“Warm” Build Time (seconds)	0.1	0.4
Indexing Time	Negligible	Several minutes
Lines of Code	319	579

Given that the vanilla version delivers the same functionality and testing capabilities without the complexity, boilerplate, or quirks, I just don’t see the advantage of using TCA.

TL;DR:

The Composable Architecture might have been useful in 2019 when SwiftUI was still immature, but Apple’s yearly updates have rendered it obsolete for most apps. TCA is a perfect example of over-engineering that actively makes code worse while claiming to make it better. If you’re transitioning to SwiftUI, focus on mastering its native tools in lieu of adopting TCA just because it feels familiar.

SwiftUI Data Flow

Tue, 26 Nov 2024 00:00:00 -0000

SwiftUI is over 5 years old. While a lot has changed since its introduction at WWDC19, the fundamentals— especially with respect to data flow—remain pretty much the same. Sure, there’s some new Observation framework syntax to learn, but Apple has done a great job at “simplifying” and “unifying” APIs while maintaining backwards compatibility.

This post is meant to serve as a reference guide for modern (iOS 17/18+) SwiftUI data flow. It ought to be particularly useful for working with LLMs that might crank out outdated code.

Data Flow

By “data flow” I mean: how data moves through an app and updates the UI. Data flow in SwiftUI is actually quite simple… as long as you keep two principles in mind: views automatically update when their data changes, and every piece of data needs a single source of truth.

Since its release Apple has produced hundreds of hours of WWDC videos on SwiftUI. If you just watch these three videos you’ll be like 90% up to speed:

Value vs Reference

I don’t want to get too in the weeds, but in order to see and understand how data flow has changed we have to differentiate between value types (📦) and reference types (🔗).

Value types like Strings and Ints and Doubles and Bools and structs and enums, create independent copies when assigned. In contrast, reference types, like classes, share the same instance.

Apple strongly recommends using value types for most cases. If we followed this advice more often I wouldn’t have even need to write this post as most of the data flow changes in SwiftUI (from iOS 13/14 to iOS 17/18+) are related to reference types!

Timeline

Legend: ✅ = current, ⛔ = avoid, 📦 = value, 🔗 = reference

iOS 13 / WWDC19

✅ 📦 @State
✅ 📦 @Binding
⛔ 🔗 ObservableObject replaced with @Observable in iOS 17+
⛔ 🔗 @Published redundant when using @Observable in iOS 17+
⛔ 🔗 @ObservedObject replaced with @Bindable in iOS 17+
⛔ 🔗 @EnvironmentObject replaced with @Environment in iOS 17+
✅ 📦 @Environment
⛔ 📦 EnvironmentKey replaced with @Entry in iOS 18+
⛔ PreviewProvider replaced with #Preview in iOS 17+

iOS 14 / WWDC20

⛔ 🔗 @StateObject replaced with @State in iOS 17+

iOS 15 / WWDC21

No significant data flow changes

iOS 16 / WWDC22

No significant data flow changes

iOS 17 / WWDC23

✅ 🔗 @Observable replaces ObservableObject
✅ 🔗 @State replaces @StateObject
✅ 🔗 @Bindable† replaces @ObservedObject
✅ 🔗 @Environment replaces @EnvironmentObject
✅ #Preview replaces PreviewProvider

iOS 18 / WWDC24

✅ 📦 @Entry replaces EnvironmentKey

Rosetta Code

Here’s a simple, but practical, example of modern data flow in SwiftUI (I’d encourage you to copy-and-paste this block of code into Xcode to experiment with the syntax for yourself):

// iOS 17/18+
import SwiftUI

extension EnvironmentValues {
    // ✅ 📦 replaces `EnvironmentKey`
    @Entry var lastRefresh: Date = .now
}

// ✅ 🔗 replaces `ObservableObject`
@Observable
class UserStore {
    var username = "max" // ✅ 🔗 replaces @Published
    var hasPremium = false // ✅ 🔗 replaces @Published

    func randomize() {
        username = String((0..<3).map { _ in "abcdefghijklmnopqrstuvwxyz".randomElement()! })
    }
}

// ✅ 🔗 replaces `ObservableObject`
@Observable
class EmojiStore {
    var emoji = "😀" // ✅ 🔗 replaces @Published

    func update() {
        emoji = String(UnicodeScalar(Int.random(in: 0x1F600...0x1F64F))!)
    }
}

struct ParentView: View {
    @State var lastRefresh: Date = .now // ✅ 📦 unchanged
    @State var userStore = UserStore() // ✅ 🔗 replaces `StateObject`
    @State var emojiStore = EmojiStore() // ✅ 🔗 replaces `StateObject`
    @State var count: Int = 0 // ✅ 📦 unchanged

    var body: some View {
        VStack(spacing: 15) {
            HStack {
                Text("Last Refresh: \(lastRefresh.formatted(date: .omitted, time: .complete))")
                Button("Refresh") { lastRefresh = .now }
            }
            HStack {
                Text("Username: \(userStore.username)")
                Button("Randomize") { userStore.randomize() }
            }
            HStack {
                Text("Has premium: \(userStore.hasPremium ? "Yes" : "No!")")
                Button("Toggle") { userStore.hasPremium.toggle() }
            }
            HStack {
                Text("Emoji: \(emojiStore.emoji)")
                Button("Update") { emojiStore.update() }
            }
            HStack {
                Text("Count: \(count)")
                Button("+1") { count += 1 }
                Button("-1") { count -= 1 }
            }
            ChildView(
                emojiStore: emojiStore, // ✅ 🔗 unchanged
                count: $count // ✅ 📦 unchanged
            )
        }
        .padding()
        .background(Rectangle().stroke(.purple, lineWidth: 1))
        .environment(\.lastRefresh, lastRefresh) // ✅ 📦 unchanged
        .environment(userStore) // ✅ 🔗 replaces `.environmentObject`
    }
}

struct ChildView: View {
    @Environment(\.lastRefresh) private var lastRefresh: Date // ✅ 📦 unchanged
    @Environment(UserStore.self) private var userStore: UserStore // ✅ 🔗 replaces `@EnvironmentObject`
    @Bindable var emojiStore: EmojiStore // ✅ 🔗 replaces `@ObservedObject`
    @Binding var count: Int // ✅ 📦 unchanged

    var body: some View {
        VStack(spacing: 15) {
            HStack {
                Text("Last Refresh: \(lastRefresh.formatted(date: .omitted, time: .complete))")
            }
            HStack {
                Text("Edit username:")
                @Bindable var userStore = userStore // ⚠️ 🔗 FRUSTRATING!
                TextField("", text: $userStore.username)
                    .textInputAutocapitalization(.never)
                    .fixedSize()
            }
            HStack {
                Text("Has premium: \(userStore.hasPremium ? "Yes" : "No!")")
                Button("Toggle") { userStore.hasPremium.toggle() }
            }
            HStack {
                Text("Emoji: \(emojiStore.emoji)")
                Button("Update") { emojiStore.update() }
            }
            HStack {
                Text("Count: \(count)")
                Button("+1") { count += 1 }
                Button("-1") { count -= 1 }
            }
        }
        .padding()
        .background(Rectangle().stroke(.mint, lineWidth: 1))
    }
}

// ✅ replaces `PreviewProvider`
#Preview("Parent") {
    ParentView()
}

// ✅ replaces `PreviewProvider`
#Preview("Child") {
    @Previewable @State var userStore = UserStore() // ✅ 🔗 replaces `StateObject`
    @Previewable @State var emojiStore = EmojiStore() // ✅ 🔗 replaces `StateObject`
    @Previewable @State var count = 10 // ✅ 📦 unchanged
    ChildView(emojiStore: emojiStore, count: $count)
        .environment(\.lastRefresh, .now) // ✅ 📦 unchanged
        .environment(userStore) // ✅ 🔗 replaces `.environmentObject`
}

And here’s the same example as above, just painted with the old, outdated syntax:

// iOS 13/14/15/16
import SwiftUI

// ⛔️ 📦 replaced with `@Entry`
private struct LastRefreshKey: EnvironmentKey {
    static let defaultValue: Date = .now
}

extension EnvironmentValues {
    // ⛔️ 📦 replaced with `@Entry`
    var lastRefresh: Date {
        get { self[LastRefreshKey.self] }
        set { self[LastRefreshKey.self] = newValue }
    }
}

// ⛔️ 🔗 replaced by `@Observable`
class UserStore: ObservableObject {
    @Published var username = "max" // ⛔️ 🔗 redundant when using `@Observable`
    @Published var hasPremium = false // ⛔️ 🔗 redundant when using `@Observable`

    func randomize() {
        username = String((0..<3).map { _ in "abcdefghijklmnopqrstuvwxyz".randomElement()! })
    }
}

// ⛔️ 🔗 replaced by `@Observable`
class EmojiStore: ObservableObject {
    @Published var emoji = "😀" // ⛔️ 🔗 redundant when using `@Observable`

    func update() {
        emoji = String(UnicodeScalar(Int.random(in: 0x1F600...0x1F64F))!)
    }
}

struct ParentView: View {
    @State private var lastRefresh: Date = .now // ✅ 📦 unchanged
    @StateObject private var userStore = UserStore() // ⛔️ 🔗 replaced with `@State`
    @StateObject private var emojiStore = EmojiStore() // ⛔️ 🔗 replaced with `@State`
    @State private var count: Int = 0 // ✅ 📦 unchanged

    var body: some View {
        VStack(spacing: 15) {
            HStack {
                Text("Last Refresh: \(lastRefresh.formatted(date: .omitted, time: .complete))")
                Button("Refresh") { lastRefresh = .now }
            }
            HStack {
                Text("Username: \(userStore.username)")
                Button("Randomize") { userStore.randomize() }
            }
            HStack {
                Text("Has premium: \(userStore.hasPremium ? "Yes" : "No!")")
                Button("Toggle") { userStore.hasPremium.toggle() }
            }
            HStack {
                Text("Emoji: \(emojiStore.emoji)")
                Button("Update") { emojiStore.update() }
            }
            HStack {
                Text("Count: \(count)")
                Button("+1") { count += 1 }
                Button("-1") { count -= 1 }
            }
            ChildView(
                emojiStore: emojiStore, // ✅ 📦 unchanged
                count: $count // ✅ 📦 unchanged
            )
        }
        .padding()
        .background(Rectangle().stroke(Color.purple, lineWidth: 1))
        .environment(\.lastRefresh, lastRefresh) // ✅ 📦 unchanged
        .environmentObject(userStore) // ⛔️ 🔗 replaced with `.environment`
    }
}

struct ChildView: View {
    @Environment(\.lastRefresh) private var lastRefresh: Date // ✅ 📦 unchanged
    @EnvironmentObject private var userStore: UserStore // ⛔️ 🔗 replaced with `@Environment`
    @ObservedObject var emojiStore: EmojiStore // ⛔️ 🔗 replaced with `@Bindable`
    @Binding var count: Int // ✅ 📦 unchanged

    var body: some View {
        VStack(spacing: 15) {
            HStack {
                Text("Last Refresh: \(lastRefresh.formatted(date: .omitted, time: .complete))")
            }
            HStack {
                Text("Edit username:")
                TextField("", text: $userStore.username)
                    .textInputAutocapitalization(.never)
                    .fixedSize()
            }
            HStack {
                Text("Has premium: \(userStore.hasPremium ? "Yes" : "No!")")
                Button("Toggle") { userStore.hasPremium.toggle() }
            }
            HStack {
                Text("Emoji: \(emojiStore.emoji)")
                Button("Update") { emojiStore.update() }
            }
            HStack {
                Text("Count: \(count)")
                Button("+1") { count += 1 }
                Button("-1") { count -= 1 }
            }
        }
        .padding()
        .background(Rectangle().stroke(Color.mint, lineWidth: 1))
    }
}

// ⛔️ replaced with `#Preview`
struct ParentView_Previews: PreviewProvider {
    static var previews: some View {
        ParentView()
    }
}

// ⛔️ replaced with `#Preview`
struct ChildView_Previews: PreviewProvider {
    static var previews: some View {
        Preview()
    }

    struct Preview: View {
        @StateObject var userStore = UserStore() // ⛔️ 🔗 replaced with `@State`
        @StateObject var emojiStore = EmojiStore() // ⛔️ 🔗 replaced with `@State`
        @State var count: Int = 10 // ✅ 📦 unchanged

        var body: some View {
            ChildView(emojiStore: emojiStore, count: $count)
                .environment(\.lastRefresh, .now) // ✅ 📦 unchanged
                .environmentObject(userStore) // ⛔️ 🔗 replaced with `.environment`
        }
    }
}

Between the Lines

As you can see, the two blocks of code are largely the same. The only real differences are in the reference type data flow. Thanks to Observation and @Observable things are a little more simple and a little better aligned with how value types have been handled since iOS 13:

1. Source of Truth

📦 @State
🔗 @State replacing @StateObject

2. Read/Write Access

📦 @Binding
🔗 @Bindable replacing @ObservedObject

3. Global Access

📦 @Environment
🔗 @Environment replacing @EnvironmentObject

Rough Edges

Observation framework is a great improvement to data flow in SwiftUI, but there are still some awkward parts. The most notable is that we need two different two-way read-write binding mechanisms (@Binding and @Bindable) instead of one unified approach. Even more frustrating is the explicit “rebinding” requirement for working with environment objects:

@Environment(UserStore.self) private var userStore
// ...
@Bindable var userStore = userStore // Required "rebinding"

Hopefully Apple unifies these binding mechanisms in the future, similar to how they’ve deftly consolidated State and Environment handling!

The Client Pattern

Wed, 20 Nov 2024 00:00:00 -0000

I recently wrapped up some work on a client project that used TCA. While I didn’t love the framework, I’m thankful for the opportunity because it introduced me to something brilliant: the Client Pattern.

The Client Pattern

The thing that I’m calling the Client Pattern is really just a simplified version of Point-Free’s DependencyClient from their swift-dependencies library. While the benefits are well articulated in this blog post, you can get most of the value with just a few lines of Swift:

import Foundation

struct MyClient {
    var fetch: @Sendable (_ value: Int) async throws -> String
}

extension MyClient {
    static let live = Self(
        fetch: { value in
            // Implementation here...
            return "Fetched value: \(value)"
        }
    )
}

func fetch(with client: MyClient = .live) async throws {
    let result = try await client.fetch(1)
    print(result)
}

No protocols. No abstractions. No third-party dependencies to manage… dependencies. Just structs and closures to make testing downstream dead simple.

A “Real” Example

To demonstrate the real power of the Client Pattern, let’s build a client for the Sunrise-Sunset API.

First, define the response type:

import Foundation

struct SuntimesResponse: Decodable {
    let results: Results

    struct Results: Decodable {
        let sunrise: Date
        let sunset: Date

        enum CodingKeys: String, CodingKey {
            case sunrise
            case sunset
        }
    }
}

Then create the client interface:

struct SuntimesClient {
    var fetchSunrise: @Sendable (_ latitude: Double, _ longitude: Double) async throws -> Date
    var fetchSunset: @Sendable (_ latitude: Double, _ longitude: Double) async throws -> Date
}

Extend the client with a .live implementation (to do the actual networking):

extension SuntimesClient {
    static let live = Self(
        fetchSunrise: { latitude, longitude in
            let result = try await Self.fetchSuntimes(latitude: latitude, longitude: longitude)
            return result.sunrise
        },
        fetchSunset: { latitude, longitude in
            let result = try await Self.fetchSuntimes(latitude: latitude, longitude: longitude)
            return result.sunset
        }
    )

    private static func fetchSuntimes(latitude: Double, longitude: Double) async throws -> SuntimesResponse.Results {
        var components = URLComponents(string: "https://api.sunrise-sunset.org/json")
        components?.queryItems = [
            URLQueryItem(name: "lat", value: "\(latitude)"),
            URLQueryItem(name: "lng", value: "\(longitude)"),
            URLQueryItem(name: "formatted", value: "0")
        ]
        guard let url = components?.url else { throw URLError(.badURL) }
        let (data, _) = try await URLSession.shared.data(from: url)
        let decoder = JSONDecoder()
        decoder.dateDecodingStrategy = .iso8601
        let response = try decoder.decode(SuntimesResponse.self, from: data)
        return response.results
    }
}

Use the @Entry macro to inject the .live implementation into the SwiftUI environment:

import SwiftUI

extension EnvironmentValues {
    @Entry var suntimesClient: SuntimesClient = .live
}

And create a view that picks up the injected client:

struct SuntimesClientView: View {
    @Environment(\.suntimesClient) var client: SuntimesClient
    @State var sunset: Date?

    var body: some View {
        VStack {
            Text(sunset?.formatted() ?? "")
            Button("Fetch Sunset") {
                Task { await fetchSunset() }
            }
        }
    }

    private func fetchSunset() async {
        do {
            sunset = try await client.fetchSunset(43.6532, -79.3832)
        } catch {
            print(error)
        }
    }
}

#Preview {
    SuntimesClientView()
}

With the Client Pattern in place, testing and previews become incredibly simple. Just define a mock client:

extension SuntimesClient {
    static let preview = Self(
        fetchSunrise: { _, _ in throw URLError(.badServerResponse) }, // Simulates an error
        fetchSunset: { _, _ in .now } // Returns the current time for sunset
    )
}

And inject it into previews:

#Preview("Client Pattern (Mock)") {
    SuntimesClientView()
        .environment(\.suntimesClient, .preview) // Injects mock client
}

Copy-and-paste

Here’s all of the code in a single block that you can copy-and-paste directly into Xcode to try out the Client Pattern for yourself:

import SwiftUI

struct SuntimesResponse: Decodable {
    let results: Results

    struct Results: Decodable {
        let sunrise: Date
        let sunset: Date

        enum CodingKeys: String, CodingKey {
            case sunrise
            case sunset
        }
    }
}

// MARK: - Client Definition
// Provides async closures for fetching sunrise and sunset data (Swift 6 ready!)
struct SuntimesClient {
    var fetchSunrise: @Sendable (_ latitude: Double, _ longitude: Double) async throws -> Date
    var fetchSunset: @Sendable (_ latitude: Double, _ longitude: Double) async throws -> Date
}

// MARK: - Live Implementation
// Implements live API calls for sunrise and sunset
extension SuntimesClient {
    static let live = Self(
        fetchSunrise: { latitude, longitude in
            let result = try await Self.fetchSuntimes(latitude: latitude, longitude: longitude)
            return result.sunrise
        },
        fetchSunset: { latitude, longitude in
            let result = try await Self.fetchSuntimes(latitude: latitude, longitude: longitude)
            return result.sunset
        }
    )

    // Helper: Makes the API call to retrieve sunrise and sunset data
    private static func fetchSuntimes(latitude: Double, longitude: Double) async throws -> SuntimesResponse.Results {
        var components = URLComponents(string: "https://api.sunrise-sunset.org/json")
        components?.queryItems = [
            URLQueryItem(name: "lat", value: "\(latitude)"),
            URLQueryItem(name: "lng", value: "\(longitude)"),
            URLQueryItem(name: "formatted", value: "0")
        ]
        guard let url = components?.url else { throw URLError(.badURL) }
        let (data, _) = try await URLSession.shared.data(from: url)
        let decoder = JSONDecoder()
        decoder.dateDecodingStrategy = .iso8601
        let response = try decoder.decode(SuntimesResponse.self, from: data)
        return response.results
    }
}

// MARK: - Preview Implementation
// Simulates sunrise and sunset fetching for previews and testing
extension SuntimesClient {
    static let preview = Self(
        fetchSunrise: { _, _ in throw URLError(.badServerResponse) }, // Simulates an error
        fetchSunset: { _, _ in .now } // Returns the current time for sunset
    )
}

// MARK: - Environment Values Extension
// Adds `@Environment(\.suntimesClient) var suntimesClient` to the SwiftUI environment (default to .live)
extension EnvironmentValues {
    @Entry var suntimesClient: SuntimesClient = .live
}

struct SuntimesClientView: View {
    @Environment(\.suntimesClient) var client: SuntimesClient
    @State var sunset: Date?

    var body: some View {
        VStack {
            Text(sunset?.formatted() ?? "")
            Button("Fetch Sunset") {
                Task { await fetchSunset() }
            }
        }
    }

    // Fetches sunset data asynchronously and updates the state
    private func fetchSunset() async {
        do {
            sunset = try await client.fetchSunset(43.6532, -79.3832)
        } catch {
            print(error)
        }
    }
}

// MARK: - Previews
#Preview("Client Pattern (Live)") {
    SuntimesClientView()
}

#Preview("Client Pattern (Mock)") {
    SuntimesClientView()
        .environment(\.suntimesClient, .preview) // Injects mock client
}

The Old Way (Don’t Do This)

For comparison, here’s the protocol-based Service Pattern I’ve previously used for app networking:

import SwiftUI
import Foundation

// MARK: - Protocol Definition
// Enables mocking and testing of sunrise/sunset fetching
protocol SuntimesServiceable {
    func fetchSunrise(latitude: Double, longitude: Double) async throws -> Date
    func fetchSunset(latitude: Double, longitude: Double) async throws -> Date
}

// MARK: - Live Service Implementation
// Implements the SuntimesServiceable protocol for actual API calls
struct SuntimesService: SuntimesServiceable {
    func fetchSunrise(latitude: Double, longitude: Double) async throws -> Date {
        let results = try await fetchSuntimes(latitude: latitude, longitude: longitude)
        return results.sunrise
    }

    func fetchSunset(latitude: Double, longitude: Double) async throws -> Date {
        let results = try await fetchSuntimes(latitude: latitude, longitude: longitude)
        return results.sunset
    }

    // Helper: Makes the API call to retrieve sunrise and sunset data
    private func fetchSuntimes(latitude: Double, longitude: Double) async throws -> SuntimesResponse.Results {
        var components = URLComponents(string: "https://api.sunrise-sunset.org/json")
        components?.queryItems = [
            URLQueryItem(name: "lat", value: "\(latitude)"),
            URLQueryItem(name: "lng", value: "\(longitude)"),
            URLQueryItem(name: "formatted", value: "0")
        ]
        guard let url = components?.url else { throw URLError(.badURL) }
        let (data, _) = try await URLSession.shared.data(from: url)
        let decoder = JSONDecoder()
        decoder.dateDecodingStrategy = .iso8601
        let response = try decoder.decode(SuntimesResponse.self, from: data)
        return response.results
    }
}

// MARK: - Mock Service for Testing
// Provides predictable behavior for testing without actual API calls
struct MockSuntimesService: SuntimesServiceable {
    func fetchSunrise(latitude: Double, longitude: Double) async throws -> Date {
        .now // Returns the current time for sunrise
    }

    func fetchSunset(latitude: Double, longitude: Double) async throws -> Date {
        throw URLError(.badServerResponse) // Simulates an error
    }
}

struct SuntimesServiceView: View {
    @State var sunrise: Date?
    private let service: SuntimesServiceable

    // Injects the dependency, defaulting to the live service
    init(service: SuntimesServiceable = SuntimesService()) {
        self.service = service
    }

    var body: some View {
        VStack {
            Text(sunrise?.formatted() ?? "")
            Button("Fetch Sunrise") {
                Task { await fetchSunrise() }
            }
        }
    }

    // Fetches sunrise data asynchronously
    private func fetchSunrise() async {
        do {
            // Warning: Swift 6 Error (Sending 'self.service' risks causing data races)
            // Sending main actor-isolated 'self.service' to nonisolated instance method 'fetchSunrise(latitude:longitude:)' 
            // risks causing data races between nonisolated and main actor-isolated uses
            sunrise = try await service.fetchSunrise(latitude: 43.6532, longitude: -79.3832)
        } catch {
            print(error)
        }
    }
}

// MARK: - Previews
#Preview("Service Pattern (Live)") {
    SuntimesServiceView(service: SuntimesService())
}

#Preview("Service Pattern (Preview)") {
    SuntimesServiceView(service: MockSuntimesService())
}

Keeping It Simple

I’m really excited about the Client Pattern going forward as it:

Removes protocol overhead
Makes testing easier with closures
Integrates well with SwiftUI
And requires less code

Against DDD

Tue, 19 Nov 2024 00:00:00 -0000

Domain-driven Design (DDD) is a software design approach that attempts to structure code to “match a domain”. It promises “adaptable”, “scalable”, and “maintainable” software by dividing apps into “bounded contexts” and by “aligning code” with “business needs”. It sounds great, in theory. But in reality, it’s a trap… especially when applied to SwiftUI.

Code as a Liability

Code isn’t an asset. It’s a liability. Every line of code is another line to test, debug, and maintain. And every line of code introduces new opportunities for new bugs. Despite the promises, DDD increases this liability by adding layers of unnecessary code and generally making a mess of everything.

Broken Promises

DDD might be okay for massive, legacy, enterprise Java systems from the early aughts. But it’s overkill for modern SwiftUI apps. Positioned as a way to make code adaptable, scalable, and maintainable, DDD simply fails on all three fronts:

Not Adaptable - DDD makes even small changes slow and tedious. Features that should take minutes can take days because changes because every layer must be re-adjusted and re-aligned.
Not Scalable - Client-side apps don’t face the same scalability challenges as backend systems. DDD’s attempt to “scale for the future” just ends up adding unnecessary cruft.
Not Maintainable - DDD buries logic under so many abstractions that understanding or updating code becomes a chore. Instead of simplifying maintenance, it creates friction and confusion.

Code Comparison

Instead of me just telling you that DDD is bad, let me show you… Consider this vanilla implementation for fetching and displaying Todos using the jsonplaceholder API:

import SwiftUI

// 1. Model

struct Todo: Codable, Identifiable {
    let id: Int
    let title: String
    let completed: Bool
}

// 2. Client (fetch todos)

struct TodoClient {
    var fetchTodos: @Sendable () async throws -> [Todo]

    static let live = Self(
        fetchTodos: {
            guard let url = URL(string: "https://jsonplaceholder.typicode.com/todos/") else { throw URLError(.badURL) }
            let (data, _) = try await URLSession.shared.data(from: url)
            let todos = try JSONDecoder().decode([Todo].self, from: data)
            return todos
        }
    )
}

// 3. View (display todos, handle errors)

struct TodoView: View {
    @State var todos: [Todo] = []
    @State var errorMessage: String? = nil
    private let client: TodoClient = .live

    var body: some View {
        List {
            ForEach(todos) { todo in
                HStack {
                    Image(systemName: todo.completed ? "checkmark.circle": "circle")
                        .foregroundStyle(todo.completed ? .green : .gray)
                    Text("\(todo.title)")
                }
            }
        }
        .navigationTitle("Todo List")
        .task { await fetch() }
        .alert("Error", isPresented: .constant(errorMessage != nil)) {
            Button("OK", role: .cancel) { errorMessage = nil }
        } message: {
            Text(errorMessage ?? "No Error")
        }
    }

    func fetch() async {
        do {
            todos = try await client.fetchTodos()
        } catch {
            errorMessage = error.localizedDescription
        }
    }
}

#Preview {
    NavigationStack {
        TodoView()
    }
}

Full DDD

And here’s what the same code might look like when you go “full DDD” (based on this popular Medium article). Honestly, don’t even try to read or understand all of it. Just notice that it’s wayyyyyy more code:

import SwiftUI

// 1. Models
// TodoDTO: The raw data from the API
// TodoDMO: A "cleaned-up" version of the data for actual app use

struct TodoDTO: Codable {
    let id: Int
    let title: String
    let completed: Bool

    func toDomainModelObject() -> TodoDMO {
        TodoDMO(id: id, label: title, isCompleted: completed)
    }
}

struct TodoDMO: Identifiable {
    let id: Int
    let label: String
    let isCompleted: Bool
}

// 2. "Data Source"
// Fetches TodoDTOs from an API, making things more complicated than they need to be

protocol TodoDataSource {
    func fetchTodos() async throws -> [TodoDTO]
}

enum TodoDataSourceError: Error {
    case badUrl
    case requestError
    case decodingError
    case statusNotOK
}

struct TodoDataSourceImplementation: TodoDataSource {
    func fetchTodos() async throws -> [TodoDTO] {
        let urlString = "https://jsonplaceholder.typicode.com/todos/"
        guard let url = URL(string: urlString) else {
            throw TodoDataSourceError.badUrl
        }
        guard let (data, response) = try? await URLSession.shared.data(from: url) else {
            throw TodoDataSourceError.requestError
        }
        guard let httpResponse = response as? HTTPURLResponse, httpResponse.statusCode == 200 else {
            throw TodoDataSourceError.statusNotOK
        }
        guard let todos = try? JSONDecoder().decode([TodoDTO].self, from: data) else {
            throw TodoDataSourceError.decodingError
        }
        return todos
    }
}

// 3. "Repository"
// Converts TodoDTOs to TodoDMOs while adding an extra layer for no obvious reason

protocol TodoRepository {
    func fetchTodos() async throws -> [TodoDMO]
}

enum TodoRepositoryError: Error {
    case decodingError
    case networkError
}

struct TodoRepositoryImplementation: TodoRepository {
    private let dataSource: TodoDataSource

    init(dataSource: TodoDataSource) {
        self.dataSource = dataSource
    }

    func fetchTodos() async throws -> [TodoDMO] {
        do {
            let todosDTO = try await dataSource.fetchTodos()
            return todosDTO.map { $0.toDomainModelObject() }
        } catch TodoDataSourceError.decodingError {
            throw TodoRepositoryError.decodingError
        } catch {
            throw TodoRepositoryError.networkError
        }
    }
}

// 4. "UseCase"
// A glorified wrapper to call the repository and handle errors in a slightly different way

protocol TodoGetUseCase {
    func execute() async -> Result<[TodoDMO], TodoGetUseCaseError>
}

enum TodoGetUseCaseError: Error {
    case networkError
    case decodingError
}

struct TodoGetUseCaseImplementation: TodoGetUseCase {
    private let repository: TodoRepository

    init(repository: TodoRepository) {
        self.repository = repository
    }

    func execute() async -> Result<[TodoDMO], TodoGetUseCaseError> {
        do {
            let todos = try await repository.fetchTodos()
            return .success(todos)
        } catch {
            return .failure(error is TodoRepositoryError ? .decodingError : .networkError)
        }
    }
}

// 5. "ViewModel"
// Mediates between the UseCase and the View, because we always need more layers!

@Observable
class TodoListViewModel {
    var todos: [TodoDMO] = []
    var errorMessage = ""
    var hasError = false

    private let useCase: TodoGetUseCase

    init(useCase: TodoGetUseCase) {
        self.useCase = useCase
    }

    func loadTodos() async {
        errorMessage = ""
        hasError = false
        let result = await useCase.execute()
        switch result {
        case .success(let todos):
            self.todos = todos
        case .failure(let error):
            self.todos = []
            errorMessage = error.localizedDescription
            hasError = true
        }
    }
}

// 6. View
// Finally, something useful: displays todos and errors, cleaning up after all the other layers

struct DDDTodoView: View {
    @Bindable var viewModel: TodoListViewModel

    init(viewModel: TodoListViewModel) {
        self.viewModel = viewModel
    }

    var body: some View {
        List {
            ForEach(viewModel.todos) { todo in
                HStack {
                    Image(systemName: todo.isCompleted ? "checkmark.circle" : "circle")
                        .foregroundColor(todo.isCompleted ? .green : .gray)
                    Text(todo.label)
                }
            }
        }
        .navigationTitle("Todo List")
        .task { await viewModel.loadTodos() }
        .alert("Error", isPresented: $viewModel.hasError) {
            Button("OK", role: .cancel) {}
        } message: {
            Text(viewModel.errorMessage)
        }
    }
}

#Preview {
    @Previewable @State var viewModel = TodoListViewModel(
        useCase: TodoGetUseCaseImplementation(
            repository: TodoRepositoryImplementation(
                dataSource: TodoDataSourceImplementation()
            )
        ))
    NavigationView {
        DDDTodoView(viewModel: viewModel)
    }
}

Both implementations do the exact same thing! Seriously, copy-and-paste both blocks into Xcode and see for yourself! All the extra layers—the repository, use case, and view model—add bloat without benefit. But what about testing? Okay. Fair. To test the vanilla version, just create a TodoClient.mock extension and inject it where needed. Done.

HackerNews

If the code comparison isn’t enough to convince you that DDD is unnecessary and actively counterproductive, these collected quotes from various HackerNews threads should:

“In my 22 years of career in software starting as a developer, I have seen DDD used only one time successfully and appropriately. All other attempts were half-baked and over-engineered messes. The problem was that over several years, it was hard to get new hires up to speed to maintain the complex code and test it.”

“DDD strikes me as the software version of Agile sometimes. The ideas and philosophy behind are good, but end up being taken as a silver bullet. I’ve seen DDD being branded together with CQRS as ‘Clean Architecture,’ which in reality turns out to be a mess of layers and separations.”

“We use DDD at the current company I work in, and to be honest, I detest it so much that sometimes it makes me wonder if I even want to continue in the programming space (been at it for 20 years). Don’t get me wrong, DDD has meaning and purpose, but some companies are applying it as a badge to be obtained instead of pondering the question: Do you really need to rewrite everything following DDD?”

“DDD seems to be one of those things where ‘what it’s meant to mean’ is very different from ‘how it’s actually practiced.’ Reading about what it’s meant to mean, it seems pretty common sense. As often implemented, however, it seems to lead to a lot of accidental complexity and a bunch of dubious usefulness abstraction layers.”

“In my experience, the most useful part of DDD is to have a common vocabulary for your projects—developers and end users should have some common terminology. For complex business domains, it is good to have a glossary and for your code to always use those words in the same way the business uses them. Other than that, most DDD concepts are a bit dated, and really oriented around JAVA/C# in the early 2000s.“

“All principles help you to model your software in a way that’s highly cohesive and loosely coupled. My suggestion: Stay away from all these design patterns. In my experience, they lead to overly complex code because of all the structure. The best structure is the most simple.”

“I thought the DDD hype had died down a bit, but I guess not. Eric Evans (inventor of DDD) has said in recent years that, unfortunately, once a team is big enough, all the invisible conceptual boundaries between domains blur and disappear. People do NOT have the discipline to do DDD correctly.“

“When you start thinking in abstractions and create abstractions in your code, either top-down or bottom-up, you will end up producing a good software design. This, to me, is the most untrue statement you could make about software. Abstractions introduce more complexity. The more complex, the less stable and maintainable your software becomes.”

“Domain-Driven Design is a poison. The book is one of the most poorly written technical books out there. There are a handful of good ideas buried in the 1,000-page unedited verbose rambling slog. The whole book should only have been 10 pages. Besides Eric Evans’s inability to write, the poison of DDD comes from locking in businesses/domain concepts into your core technology, making them inflexible and making it difficult for the business to iterate on new ideas.”

“The biggest flaw of DDD I’ve run into is there’s no emphasis on when not to use it. There’s no mention that over-coding business rules into modules and services locks you into business processes that are slow or impossible to update. There’s no mention that most times you want to build services that offer platform capabilities, not focus on what ‘domain’ they fall into. Never mind that ‘domain’ is basically undefined and can mean many different concepts and different types of concepts.”

Go for Simple

SwiftUI thrives on simplicity. Not endless layers of abstractions. Instead of wasting time on “domains”, “repositories”, and “use cases”. Let’s focus on delivering features quickly with less code.

Grug Brained SwiftUI

Sat, 16 Nov 2024 00:00:00 -0000

A SwiftUI tribute to the timeless wisdom of “The Grug Brained Developer”, a guide that shows simplicity is not just a choice but a way of life.

swiftui

one day grug find swiftui

grug like swiftui

swiftui good. swiftui simple. swiftui flow like river

but some dev stack rock in river, make waterfall of sad

grug think “why fight river with rock???”

“why make simple thing complex???”

grug say “stop fight river. let river flow!”

but some dev not listen

some dev throw rock in river again and again

mvvm

grug see dev who fight river also use mvvm

grug not big fan. grug know mvvm not always fit swiftui

but if mvvm there, grug stay quiet, for shared goal

until grug see nested view model

grug think “why nest view model like angry babushka doll??!”

grug say “view model not nest. view model hold state for one thing, not many things!”

nest not just throw rock in river but build dam of despair

llm

grug see dev who fight river also use llm for swiftui

grug think “llm fine. good tool, like sharp axe”

but some dev swing axe bad

use llm code, no check if good

like build boat with bad wood

boat float, then sink

grug say “llm not magic”

“tell llm wrong thing, like help uncle jack off the horse“

“llm give handjob to horse instead”

grug say “llm only good if dev know good llm code from bad”

boat sink if not

complexity

grug know all swiftui struggles come from one thing: complexity

complexity creep in slow

grow like debris in river

at first seem fine

then river clog

water stop

code become swamp

grug only want swiftui stay like river and not turn to swamp

Code Philosophy

Fri, 15 Nov 2024 00:00:00 -0000

John Ousterhout’s A Philosophy of Software Design is a masterpiece of software engineering wisdom. Here are some of the most impactful quotes from the book, organized by theme.

Complexity

Complexity is the heart of most software engineering challenges.

Understanding Complexity: “If a software system is hard to understand and modify, then it is complicated; if it is easy to understand and modify, then it is simple.”
Change Amplification: “The first symptom of complexity is that a seemingly simple change requires code modifications in many different places.”
Cognitive Load: “The second symptom of complexity is cognitive load, which refers to how much a developer needs to know in order to complete a task.”
Incremental Complexity: “Complexity is incremental: you have to sweat the small stuff.”
Accumulating Complexity: “Over time, complexity accumulates, and it becomes harder and harder for programmers to keep all of the relevant factors in their minds as they modify the system.”

Code Design

Good design is all about making the right tradeoffs and choosing the appropriate abstractions.

Simplicity: “It’s more important for a module to have a simple interface than a simple implementation.”
Write Twice; Commit Once: “Designing software is hard, so it’s unlikely that your first thoughts about how to structure a module or system will produce the best design.”
Abstractions: “When designing methods, the most important goal is to provide clean and simple abstractions.”
Deep Modules: “The best modules are deep: they have a lot of functionality hidden behind a simple interface.”
Information Hiding: “Information hiding makes it easier to evolve the system.”
Avoiding Config Parameters: “You should avoid configuration parameters as much as possible.”

DX (Developer Experience)

The humans in the software development loop are just as important as the “loops” themselves.

Optimize for Legibility: “Software should be designed for ease of reading, not ease of writing.”
Reduce Suffering: “Most modules have more users than developers, so it is better for the developers to suffer than the users.”
Comments and Clarity: “The guiding principle for comments is that comments should describe things that aren’t obvious from the code.”
Obviousness in Code: “If someone reading your code says it’s not obvious, then it’s not obvious, no matter how clear it may seem to you.”
Choosing Good Names: “Choosing a mediocre name for a particular variable… will have a significant impact on complexity and manageability.”

Taken together these quotes stress that software is as much about managing complexity as it is about writing code.

SwiftUI is not UIKit

Thu, 14 Nov 2024 00:00:00 -0000

Programming basically boils down to two things: data flow and naming stuff. Simple in theory. Super easy to complicate in practice. To illustrate the point let’s look at some iOS code that updates an emoji on a “parent” view from a “child” view action.

Boilerplate UIKit

In UIKit this relatively simple task becomes a whole sad song and dance:

import UIKit

class ParentViewController: UIViewController {
    var labelText = "🐶" {
        didSet {
            label.text = "Label: \(labelText)"
            childViewController?.updateLabel(with: labelText)
        }
    }

    private let label = UILabel()
    private let childContainer = UIView()
    private var childViewController: ChildViewController?

    override func viewDidLoad() {
        super.viewDidLoad()
        setupUI()
    }

    private func setupUI() {
        view.backgroundColor = .white
        view.layer.borderColor = UIColor.orange.cgColor
        view.layer.borderWidth = 1
        view.layoutMargins = UIEdgeInsets(top: 10, left: 10, bottom: 10, right: 10)
        let parentStack = UIStackView(arrangedSubviews: [
          createParentLabelStack(), childContainer
        ])
        parentStack.axis = .vertical
        parentStack.spacing = 8
        parentStack.translatesAutoresizingMaskIntoConstraints = false
        view.addSubview(parentStack)
        NSLayoutConstraint.activate([
            parentStack.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            parentStack.centerYAnchor.constraint(equalTo: view.centerYAnchor),
            childContainer.heightAnchor.constraint(equalToConstant: 100),
            childContainer.widthAnchor.constraint(equalToConstant: 150)
        ])
        let childVC = ChildViewController()
        childVC.labelText = labelText
        childVC.onUpdateText = { [weak self] newText in
            self?.labelText = newText
        }
        addChild(childVC)
        childContainer.addSubview(childVC.view)
        childVC.view.frame = childContainer.bounds
        childVC.didMove(toParent: self)
        childViewController = childVC
        childContainer.layoutMargins = UIEdgeInsets(
            top: 10, left: 10, bottom: 10, right: 10
        )
    }

    private func createParentLabelStack() -> UIView {
        let parentLabel = UILabel()
        parentLabel.text = "Parent"
        let stack = UIStackView(arrangedSubviews: [parentLabel, label])
        stack.axis = .vertical
        stack.alignment = .center
        stack.spacing = 4
        stack.layer.borderColor = UIColor.red.cgColor
        stack.layer.borderWidth = 1
        stack.layoutMargins = UIEdgeInsets(top: 10, left: 10, bottom: 10, right: 10)
        stack.isLayoutMarginsRelativeArrangement = true
        label.text = "Label: \(labelText)"
        label.textAlignment = .center
        return stack
    }
}

class ChildViewController: UIViewController {
    var labelText = "🐶" {
        didSet {
            label.text = "Label: \(labelText)"
        }
    }

    var onUpdateText: ((String) -> Void)?
    private let label = UILabel()
    private let updateButton = UIButton(type: .system)

    override func viewDidLoad() {
        super.viewDidLoad()
        setupUI()
    }

    private func setupUI() {
        view.layer.borderColor = UIColor.green.cgColor
        view.layer.borderWidth = 1
        view.layoutMargins = UIEdgeInsets(top: 10, left: 10, bottom: 10, right: 10)
        label.text = "Label: \(labelText)"
        label.textAlignment = .center
        updateButton.setTitle("Update", for: .normal)
        updateButton.addTarget(self, action: #selector(handleUpdateButtonTap), for: .touchUpInside)
        let childLabel = UILabel()
        childLabel.text = "Child"
        let stack = UIStackView(arrangedSubviews: [childLabel, label, updateButton])
        stack.axis = .vertical
        stack.alignment = .center
        stack.spacing = 4
        stack.translatesAutoresizingMaskIntoConstraints = false
        stack.isLayoutMarginsRelativeArrangement = true
        view.addSubview(stack)
        NSLayoutConstraint.activate([
            stack.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            stack.centerYAnchor.constraint(equalTo: view.centerYAnchor)
        ])
    }

    @objc private func handleUpdateButtonTap() {
        labelText = "🐈"
        onUpdateText?(labelText)
    }

    func updateLabel(with text: String) {
        labelText = text
    }
}

// Only required for Xcode Previews
import SwiftUI

struct ParentViewControllerWrapper: UIViewControllerRepresentable {
    @Binding var text: String

    func makeUIViewController(context: Context) -> ParentViewController {
        let parentVC = ParentViewController()
        parentVC.labelText = text
        return parentVC
    }

    func updateUIViewController(_ uiViewController: ParentViewController, context: Context) {
        uiViewController.labelText = text
    }
}

#Preview {
    @Previewable @State var text = "🐶"
    ParentViewControllerWrapper(text: $text)
        .frame(width: 200, height: 220)
}

Seriously?! All of that to change an emoji from “🐶” to “🐈”!

“SwiftUIKit”

Many seasoned UIKit developers transitioning to SwiftUI treat the framework as “UIKit with different syntax.” The result? Bloated SwiftUI code that looks like this:

import SwiftUI

// Unnecessary coordinator pattern
class ParentCoordinator {
    // Unneccessary nesting
    weak var parentViewModel: ParentViewModel?
    var childCoordinator: ChildCoordinator?

    init(parentViewModel: ParentViewModel) {
        self.parentViewModel = parentViewModel
        self.childCoordinator = ChildCoordinator(childViewModel: parentViewModel.childViewModel)
        self.childCoordinator?.delegate = self
    }
}

// Unneccessary delegate pattern
extension ParentCoordinator: ChildCoordinatorDelegate {
    func childCoordinatorDidUpdateLabelText(_ newText: String) {
        parentViewModel?.labelText = newText
    }
}

// Unneccessary protocol
protocol ChildCoordinatorDelegate: AnyObject {
    func childCoordinatorDidUpdateLabelText(_ newText: String)
}

// Unneccessary child coordinator
class ChildCoordinator {
    // Unneccessary nesting
    weak var childViewModel: ChildViewModel?
    weak var delegate: ChildCoordinatorDelegate?

    init(childViewModel: ChildViewModel) {
        self.childViewModel = childViewModel
    }

    // Unnecessary @MainActor
    @MainActor func updateText() {
        childViewModel?.updateText()
        delegate?.childCoordinatorDidUpdateLabelText(childViewModel!.labelText)
    }
}

// Unneccessary view model
@Observable
class ParentViewModel {
    var labelText: String
    // Unneccessary nesting
    var childViewModel: ChildViewModel
    var coordinator: ParentCoordinator?

    init(labelText: String = "🐶") {
        self.labelText = labelText
        self.childViewModel = ChildViewModel(labelText: labelText)
        self.coordinator = ParentCoordinator(parentViewModel: self)
    }
}

// Unneccessary view model
@Observable
class ChildViewModel {
    var labelText: String

    init(labelText: String) {
        self.labelText = labelText
    }

    // Unneccessary @MainActor
    @MainActor func updateText() {
        labelText = "🐈"
    }
}

// Overly complex view hierarchy
struct ParentView: View {
    @Bindable var viewModel: ParentViewModel

    init() {
        let viewModel = ParentViewModel()
        self.viewModel = viewModel
    }

    var body: some View {
        VStack {
            VStack {
                Text("Parent")
                Text("Label: \(viewModel.labelText)")
            }
            .padding()
            .background(Rectangle().stroke(Color.red))
            ChildView(viewModel: viewModel.childViewModel, coordinator: viewModel.coordinator!.childCoordinator!)
        }
        .padding()
        .background(Rectangle().stroke(Color.orange))
    }
}

// Child view with direct connection to the coordinator
struct ChildView: View {
    @Bindable var viewModel: ChildViewModel
    var coordinator: ChildCoordinator

    var body: some View {
        VStack {
            Text("Child")
            Text("Label: \(viewModel.labelText)")
            Button(action: {
                coordinator.updateText()
            }) {
                Text("Update")
            }
        }
        .padding()
        .background(Rectangle().stroke(Color.green))
    }
}

#Preview {
    ParentView()
}

Coordinators, delegates, nested view models… and enough boilerplate to sink a ship!

Vanilla SwiftUI

Here’s the exact same problem solved with just a few lines of SwiftUI:

import SwiftUI

struct ParentView: View {
    // Single source of truth using @State
    @State private var text = "🐶"

    // Simple, declarative view hierarchy
    var body: some View {
        VStack {
            VStack {
                Text("Parent")
                Text("Label: \(text)")
            }
            .padding()
            .background(Rectangle().stroke(.red))
            ChildView(text: $text)
        }
        .padding()
        .background(Rectangle().stroke(.orange))
    }
}

struct ChildView: View {
    // Direct connection to parent state using @Binding
    @Binding var text: String

    var body: some View {
        VStack {
            Text("Child")
            Text("Label: \(text)")
            Button {
                text = "🐈"
            } label: {
                Text("Update")
            }
        }
        .padding()
        .background(Rectangle().stroke(.green))
    }
}

#Preview {
    ParentView()
}

No coordinators. No delegates. No view models. Just @State and @Binding doing what they were designed to do. SwiftUI is able to handle all of the complexity that had to be manually orchestrated in UIKit.

MVVM

Some might complain, but what about MVVM? Well, if you must (a topic for another time) use MVVM with SwiftUI here’s what that might look like:

import SwiftUI

@Observable
class ViewModel {
    // Single source of truth at the model layer
    var text = "🐶"

    func update() {
        text = "🐈"
    }
}

struct ParentView: View {
    // View owns the model instance
    @State var viewModel = ViewModel()

    var body: some View {
        VStack {
            VStack {
                Text("Parent")
                Text("Label: \(viewModel.text)")
            }
            .padding()
            .background(Rectangle().stroke(.red))
            ChildView()
        }
        .padding()
        .background(Rectangle().stroke(.orange))
        .environment(viewModel)
    }
}

struct ChildView: View {
    // Child accesses shared model through environment
    @Environment(ViewModel.self) var viewModel: ViewModel

    var body: some View {
        VStack {
            Text("Child")
            Text("Label: \(viewModel.text)")
            Button {
                viewModel.update()
            } label: {
                Text("Update")
            }
        }
        .padding()
        .background(Rectangle().stroke(.green))
    }
}

#Preview {
    ParentView()
}

At least the thing given the ViewModel name this time is actually a view model!

The Bottom Line

SwiftUI isn’t just a new UI framework—it’s a completely different way of thinking. A brand new way of building. UIKit patterns worked well… in UIKit. Forcing them into SwiftUI only creates complexity.

How to Win Fantasy Hockey

Sat, 28 Sep 2019 00:00:00 -0000

How to Win Fantasy Hockey

A recipe for using python to scrape player projection data and build a value over replacement ranking model

Jupyter notebook for this blog post available here

№1 Identify a data source (CBS Sports is a good option)

№2 get the html data with gazpacho

from gazpacho import get

position = 'F'

base = f'https://www.cbssports.com/fantasy/hockey/stats'
url = f'{base}/{position}/2019/restofseason/projections/'

html = get(url)

№3 Pass the captured html to a Soup parser

from gazpacho import Soup

soup = Soup(html)

№4 find the html tags that contain player projection data

# HTML: <tr class="TableBase-bodyTr ">
rows = soup.find('tr', {'class': 'TableBase-bodyTr '})

№5. Capture a single row (and inspect for good measure)

row = rows[0]

№6 Use find to grab those tags that map to player name, position, and projected Fantasy Points

# name
row.find('span', {'class': 'CellPlayerName--long'}).find('a').text

# position
(row.find('span', {'class': 'CellPlayerName--long'})
     .find('span', {'class': 'CellPlayerName-position'}).text
)

# points
float(row.find('td', {'class': 'TableBase-bodyTd'})[1].text)

361.1

№7 Wrap these find operations into a function

row = rows[0]

def parse_row(row):
    meta = row.find('span', {'class': 'CellPlayerName--long'})
    try:
        name = meta.find('a').text
    except AttributeError:
        name = meta.text
    position = meta.find('span', {'class': 'CellPlayerName-position'}).text
    points = float(row.find('td', {'class': 'TableBase-bodyTd'})[1].text)
    return name, position, points

parse_row(row)

('Nikita Kucherov', 'RW', 361.1)

№8 Make sure that the function works for all the captured rows

players = []
for row in rows:
    try: 
        players.append(parse_row(row))
    except AttributeError:
        pass

players[-2:]

[('Brandon Pirri', 'LW', 165.4), ('Pavel Buchnevich', 'RW', 164.9)]

№9 Bundle up the logic so that it can be applied to multiple pages

def scrape_position(position):
    base = f'https://www.cbssports.com/fantasy/hockey/stats'
    url = f'{base}/{position}/2019/restofseason/projections/'
    html = get(url)
    soup = Soup(html)
    rows = soup.find('tr', {'class': 'TableBase-bodyTr '})
    data = []
    for row in rows:
        try: 
            data.append(parse_row(row))
        except AttributeError:
            pass
    return data

№10 Scrape each page that contains player projection data

# F for Forwards
# D for Defence
# G for goalies

import time

data = []
for position in ['F', 'D', 'G']:
    d = scrape_position(position)
    data.extend(d)
    time.sleep(1)

№11 Stuff the captured data into a pandas DataFrame

import pandas as pd

df = pd.DataFrame(data, columns=['name', 'position', 'points'])
df.sample(5)

name position  points
171     Mike Matheson        D    92.0
60       David Perron       RW   196.9
86        Tomas Tatar       LW   176.1
190  Dennis Cholowski        D    83.0
257    Thatcher Demko        G    89.0

№12 Calculate the value over player replacement score for each player

pool_size = 8
starters = {'C': 1, 'LW': 1, 'RW': 1, 'D': 2, 'G': 1}

for position, slots in starters.items():
    replacement = (
        df[df['position'] == position]
        .sort_values('points', ascending=False)
        .head(slots * pool_size)
        ['points']
        .mean()
    )
    df.loc[df['position'] == position, 'vorp'] = df['points'] - replacement

№13 Re-rank and draft players according to their VORP rank

df['rank'] = df['vorp'].rank(method='average', ascending=False)
df.sort_values('rank').set_index('rank').head(20)

name position  points      vorp
rank                                               
1.0      Nikita Kucherov       RW   361.1  75.38750
2.0       Leon Draisaitl       LW   333.1  58.51250
3.0       Connor McDavid        C   337.1  45.13750
4.0   Andrei Vasilevskiy        G   433.9  41.26250
5.0          Brent Burns        D   225.1  34.09375
6.0        Brad Marchand       LW   302.1  27.51250
7.0         Patrick Kane       RW   312.7  26.98750
8.0     Dustin Byfuglien        D   216.7  25.69375
9.0        Mark Giordano        D   210.2  19.19375
10.0       Morgan Rielly        D   208.0  16.99375
11.0        John Carlson        D   207.0  15.99375
12.0         Kris Letang        D   205.2  14.19375
13.0        Tyson Barrie        D   203.6  12.59375
14.0      Mikko Rantanen       RW   297.2  11.48750
15.0    Nathan MacKinnon        C   302.8  10.83750
16.0       Alex Ovechkin       LW   283.9   9.31250
17.0   Jordan Binnington        G   393.5   0.86250
18.0          Torey Krug        D   191.8   0.79375
19.0      Steven Stamkos        C   292.3   0.33750
20.0         Tuukka Rask        G   392.7   0.06250

№14 Profit

mummify 1.0 Release

Mon, 23 Sep 2019 00:00:00 -0000

mummify just earned it’s 1.0 release!

It’s kind of scary to kick software out of 0.x and label it “production ready”. But after using it for a year, it feels ready…

Origin

I started writing mummify in early 2018 and debuted it at AnacondaCON that year in a talk titled ‘Data Engineering for Data Scientists’. Though the presentation was mostly tongue-in-cheek, the talk and the package was well received.

A couple of cheeky slides from AnacondaCON

Since then I’ve used mummify in several machine learning projects and have heard from several more people who use it regularly.

About

What is mummify? mummify is git + logging for ML…

Basically, the package takes care of git and logging for your machine learning project so that you can focus on what’s important.

Usage

mummify is dead simple to use. Just import it at the top and add mummify.log(<string>) at the bottom:

import mummify

...

model = KNeighborsClassifier()
model.fit(X_train, y_train)
accuracy = model.score(X_test, y_test)

...

mummify.log(f'Test Accuracy: {accuracy}')

When enabled mummify will save model snapshots and log performance after each run.

New

What’s new in 1.0? Honestly, nothing. Though I rewrote the entire backend to leverage subprocess.run instead of older functions, the API remains the same.

yass (Yet Another Static Site)

Thu, 19 Sep 2019 00:00:00 -0000

I just finished* writing the static site generator for this blog. And I’m actually kind of excited about it! Though creating yet another static site engine is probably—no definitely—overkill, none of generators I looked at and played with met my exact needs…

I knew that I wanted something super lightweight. Something that could seamlessly handle markdown files and Jupyter notebooks. And something that was built in Python. So I built my own thing.

Right now, the main chunk of code that powers this website is sitting at about 100 lines. Where, importantly, most of the heavy lifting within that file is delegated to jinja2, markdown2, and nbconvert. Beyond these wonderful packages, I’m using Bootstrap, Pygments and GitHub Pages to pull everything together.

And it really does all pull together. For example, this page is generated by this notebook. All I had to do to publish was type make publish into a terminal.

With my new bespoke static site generator (which, I guess I should just call yass at this point?) I can handle DataFrames:

import pandas as pd

df = pd.DataFrame({
    'artist': ['Tame Impala', 'Childish Gambino', 'The Knocks'],
    'listens': [8_456_831, 18_185_245, 2_556_448]
})

df

artist   listens
0       Tame Impala   8456831
1  Childish Gambino  18185245
2        The Knocks   2556448

Print statements and flashy outputs (shameless chart plug):

from chart import bar

print('\nGood shit...\n')
bar(df.listens, df.artist, width=20, mark='🔊')

Good shit...

     Tame Impala: 🔊🔊🔊🔊🔊🔊🔊🔊🔊           
Childish Gambino: 🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊🔊
      The Knocks: 🔊🔊🔊

Matplotlib graphs (harder than it looks, but that’s the point):

import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline

np.random.seed(42)
N = 500
x = np.random.normal(10, 5, size=N)
y = 2 * x + np.random.normal(0, 5, size=N)

plt.figure(figsize=(6, 4))
plt.scatter(x, y, c='r', alpha=1/5);

And embedded images:

I’ve ostensibly removed all of the barriers that historically prevented me from updating my blog regularly over the past couple of years. So expect me to write more? Hopefully? We’ll see… ^{*nothing is ever finished}

Kindle clippings.txt with Python

Sat, 23 Dec 2017 00:00:00 -0000

Exactly a year ago I posted Kindle clippings.txt with R. Since then things have changed… I’m a Pythonista now! Consequently, I thought it would be fun to update that post and parse highlights with 3.6+ and pandas. Janky, but it works:

import pandas as pd

txt = """Sourdough (Robin Sloan)
- Your Highlight on page 187 | Location 2853-2855 | Added on Tuesday, October 2, 2017 8:47:09 PM

The world is going to change, I think—slowly at first, then faster than anyone expects.
==========
Sapiens (Yuval Noah Harari)
- Your Highlight on page 196 | Location 2996-2997 | Added on Tuesday, October 3, 2017 8:51:09 PM

Evolution has made Homo sapiens, like other social mammals, a xenophobic creature.
==========
Life 3.0 (Max Tegmark)
- Your Highlight on page 75 | Location 1136-1137 | Added on Wednesday, October 11, 2017 6:00:15 PM

In short, computation is a pattern in the spacetime arrangement of particles
==========
"""

with open('clippings.txt', 'w', encoding='utf-8-sig') as f:
    f.write(txt)

with open('clippings.txt', 'r', encoding='utf-8-sig') as f:
    contents = f.read().replace(u'\ufeff', '')
    lines = contents.rsplit('==========')
    store = {'author': [], 'title': [], 'quote': []}
    for line in lines:
        try:
            meta, quote = line.split(')\n- ', 1)
            title, author = meta.split(' (', 1)
            _, quote = quote.split('\n\n')
            store['author'].append(author.strip())
            store['title'].append(title.strip())
            store['quote'].append(quote.strip())
        except ValueError:
            pass

df = pd.DataFrame(store)
print(df.to_csv(index=False, encoding='utf-8-sig'))
# author,quote,title
# Robin Sloan,"The world is going to change, I think—slowly at first, then faster than anyone expects.",Sourdough
# Yuval Noah Harari,"Evolution has made Homo sapiens, like other social mammals, a xenophobic creature.",Sapiens
# Max Tegmark,"In short, computation is a pattern in the spacetime arrangement of particles",Life 3.0

Right now I’m 49 books deep. It’s crunch time, but I can see the end! Look for my annual 52 Quotes post in a couple of days!

Animated Spirals

Mon, 20 Feb 2017 00:00:00 -0000

Ed Hawkins’ Global Temperature Spiral is my new favourite visualization.

It’s powerful, compelling, and super tangible.

I wanted to apply the spiral to my own data, so I got janky with ggplot2 and figured out how to do it.

Here’s my own spiral with StatsCan data on tourists visits to Canada 🍁:

(Side Note: Y U no come to Canada N E more? It’s our Sesquicentennial this year, the CAD is dirt cheap, and we have good beer!)

Spirals look super cool, but really, there’s not much to them. You can effectively spin up your own in 4 lines of R:

ggplot(df, aes(x = month, y = value)) +
    geom_line(aes(colour = year, group = year)) +
    coord_polar(theta = "x") +
    scale_x_continuous(breaks = seq(1, 12, 1))

The hard part is getting the data to squish into those four lines.

This is how I made the StatsCan data all squishy:

library(tidyverse)
library(stringr)

URL <- str_c("https://raw.githubusercontent.com/maxhumber",
    "/maxhumber.com/master/assets/data/tourists.csv")

df <- read_csv(URL) %>%
    select(Ref_Date, value = Value) %>%
    separate(Ref_Date, into = c("year", "month"), sep = "/") %>%
    mutate(year = as.numeric(year), month = as.numeric(month)) %>%
    filter(year >= 2000) %>% 
    group_by(year) %>%
    # add zeroth month with fill to make spiral actually spiral
    complete(month = seq(0, 12, 1)) %>%
    ungroup() %>%
    fill(value, .direction = "down") %>%
    arrange(year, month) %>%
    drop_na() %>%
    # 2 data points in first frame for geom_line to work
    mutate(frame = lag(row_number())) %>%
    fill(frame, .direction = "up")

The trick, I figured out, is to split out and separate the months from the years and create a dummy linking month so that the spiral can actually spiral. If you’re playing along at home, I’d recommend running things one line at a time to see the intermediate wrangling steps.

After the data is wrangled, you can spice up the original 4 lines and wrap it in a function to pass to animation::saveGIF.

library(viridis)
library(animation)

draw_spiral <- function(i=1) {

    # plot sequence
    p <- df %>% 
        filter(frame <= i) %>% 
        ggplot(aes(x = month, y = value))

    # add outline and gridlines with labels
    p <- p +
        geom_rect(aes(
            xmin = 0, ymin = min(df$value, na.rm = TRUE) - 2e6, 
            xmax = 12, ymax = max(df$value, na.rm = TRUE) + 2e6),
            fill = "#000000") + 
        # 1M
        geom_hline(yintercept = 1e6, color = "red") + 
        annotate("label", x = 0, y = 1e6, color = "red", 
            fill = "#000000", label = "1M", size = 3) + 
        # 7M
        geom_hline(yintercept = 7e6, color = "red") + 
        annotate("label", x = 0, y = 7e6, color = "red", 
            fill = "#000000", label = "7M", size = 3)

    # plot actual data
    p <- p +
        geom_line(aes(colour = year, group = year))

    # year label
    p <- p +
        geom_text(
            data = ( df %>% filter(frame == i) ), aes(
            label = year, x = 6, y = min(df$value, na.rm = TRUE) - 2e6),
            color = "#FFFFFF")

    # coordinate and scale systems
    p <- p +
        coord_polar(theta = "x") +
        scale_y_continuous(
            limits = range(df$value, na.rm = TRUE) + c(-2e6, 2e6)) +
        scale_x_continuous(
            # hack out the zeroth month
            breaks = seq(1, 12, 1),
            labels = c(
                "Jan", "Feb", "Mar", "Apr", 
                "May", "Jun", "Jul", "Aug", 
                "Sep", "Oct", "Nov", "Dec"))

    # theme and formats
    p <- p +
        scale_color_viridis(limits = range(df$year), begin = 0, end = 1) + 
        labs(title = "Total Visitors to Canada (2000 - 2016)", 
             caption = "@maxhumber") + 
        theme(
            legend.position = "none",
            panel.background = element_rect(fill = "#313131"),
            plot.background = element_rect(fill = "#313131"),
            plot.title = element_text(color = "#FFFFFF", hjust = 0.5, size = 16),
            plot.caption = element_text(color = "#FFFFFF", hjust = 1, size = 10),
            panel.grid = element_blank(), 
            axis.text.y = element_blank(),
            axis.text.x = element_text(
                color = "#FFFFFF", size = 12, 
                angle = seq(-30, -360, length.out = 12)),
            axis.title = element_blank(), 
            axis.ticks = element_blank())

    print(p)
}

# draw_spiral(i=219)

And, finally, you can wrap the function with an animation snippet to make the spiral actually spiral. (I initially tried to use gganimate but things started to get weird (@drob), so I fell back on vanilla animation):

saveGIF({
    for (i in 1:(nrow(df)-1)) {
        draw_spiral(i)
    }},
    movie.name = "canada.gif", 
    interval = 1/12, 
    ani.width = 600, 
    ani.height = 600
)

Hope to see your spirals soon 🌀. And I hope to see you in July 🍁!

If you’ve enjoyed this post you might also like what I’ve done on:

Canadian Tile Grid Maps

Wed, 15 Feb 2017 00:00:00 -0000

I love the tile grid map:

Source

I think they’re elegant, easy to read, and great for data that are population agnostic.

I live in Canada, though. And it sucks (well, no it doesn’t!) because I’ve only ever seen tile grids for America.

I decided that Canada deserved one.

So, here is the tile grid map template for all you Canadian useRs:

library(tidyverse)
library(viridis)

# LOCATION DATAFRAME

canada <- tribble(
    ~province, ~code, ~x, ~y,
    "Yukon", "YK", 1, 3,
    "British Columbia", "BC", 1, 2, 
    "Northwest Territories", "NW", 2, 3, 
    "Alberta", "AB", 2, 2, 
    "Nunavut", "NU", 3, 3, 
    "Saskatchewan", "SK", 3, 2,
    "Manitoba", "MB", 4, 2, 
    "Ontario", "ON", 5, 1,
    "Quebec", "QC", 6, 2, 
    "Prince Edward Island", "PE", 7, 3, 
    "New Brunswick", "NB", 7, 2,
    "Newfoundland and Labrador", "NL", 8, 3, 
    "Nova Scotia", "NS", 8, 1
)

# YOUR DATA HERE
# just make sure province = province

df <- tibble(
    province = c("Ontario", "British Columbia", "Alberta", "Manitoba", "Nova Scotia"), 
    `Fake Data` = c(10, 4, 3, 6, 0)
)

# TILE GRID MAP

canada %>% 
    left_join(df, by = "province") %>% 
    ggplot(aes(x, y)) + 
    geom_tile(aes(fill = `Fake Data`)) + 
    geom_text(aes(label = code), color = "white") + 
    coord_fixed(ratio = 1) + 
    theme(
        panel.background = element_blank(),
        panel.grid = element_blank(), 
        axis.title = element_blank(), 
        axis.text = element_blank(), 
        axis.ticks = element_blank()) + 
    scale_fill_viridis(na.value = "#E1E1E1", option = "D", begin = 0.2, end = 0.8)

Starry Night Plots

Sun, 05 Feb 2017 00:00:00 -0000

I think good data science reads like a good story. In that it flows. Has an arc. And is compelling.

But data science has a dirty secret. For every piece that works, there are about nine others that didn’t. Nine other stories that look like they were typed up by a monkey with a typewriter (and in a finite amount of time!)

The problem with this problem is that unless you did the work you never get to see the monkey scripts. The stuff that never worked. The stuff that got thrown out.

This creates, I think, unrealistic expectations about how data science gets done. Because every tutorial, textbook, lesson, workshop, talk, and blog post about data science just works. And we forget that this stuff has been designed and vetted to work. Most real world stuff doesn’t just work…

I initially wanted this post to be about the Super Bowl. To build some simple toy model on “Points For/Against” and push it through a new plot that I’ve been working on. But nothing was working. The data and my toy model told a story that literally made zero sense.

But then it struck me. This is perfect. I was able to catch and diagnose a garbage model exactly because of the plot I wanted to talk about!

So. Here’s my monkey script and the “Starry Night Plot” that helped me throw it out:

SETUP

library(tidyverse)
library(viridis)

DATA

df <- read_csv(paste0(
    "https://raw.githubusercontent.com/maxhumber/",
    "maxhumber.com/master/assets/data/superbowl_data.csv"))

These data are scraped from pro-football-reference.com with purrr and rvest. If you want to learn more about webscraping within the tidverse I have a quick post on the process here.

MODEL

I was trying to build a toy model to predict who might win Super Bowl LI. I don’t know much about the NFL, but I thought that I might be able to do something with “Season Average Points For” and “Season Average Points Against”. Basically, I wanted to tease out the relative importance of Defense and Offense for past Super Bowl contenders:

mod1 <- glm(
    win ~ pf + pa,
    family = "binomial",
    data = df)

# summary(mod1)

If you run summary(mod1) you can see that the p-values aren’t traditionally “significant”, however, pf hits 0.1 which gets you a little . if you’re “star-gazing” the printout.

Pumping the model through a prediction grid of all likely/possible points combinations:

pred_grid <- expand.grid(
    pf = 0:40, 
    pa = 0:40)

pred_grid$prob <- predict(
    mod1, newdata = pred_grid, type = "response")

And the putting that grid through a Tile Plot gets you:

pred_grid %>% 
    ggplot(aes(x = pf, y = pa)) + 
    geom_tile(aes(fill = prob))

Sweet. Looks alright. But what does it mean? It’s kind of hard to tell…

To improve the interpretability of geom_tile I added a contour layer and made the breaks hit the same sequence.

br <- 0.2

pred_grid %>% 
    ggplot(aes(x = pf, y = pa)) + 
    geom_tile(aes(fill = prob)) + 
    geom_contour(aes(z = prob), color = "white", 
        lty = 2, binwidth = br) + 
    scale_fill_continuous(
        breaks = seq(0, 1, br),
        labels = scales::percent_format())

Now I can see that my dumb toy model thinks that “Points For” is more predictive than “Points Against”. And that “Season Average Points For” is a bad thing when it comes to winning the Super Bowl.

Basically, if I put Atlanta and New England through this model the better team on paper would be expected to lose the Super Bowl (lol).

I wanted to figure out why my toy model was behaving so strangely, so I added in the underlying data through a couple of geom_point layers (Os for actual wins, Xs for actual losses) and a bit of polish:

br <- 0.2

pred_grid %>% 
    ggplot(aes(x = pf, y = pa)) + 
    geom_tile(aes(fill = prob)) + 
    geom_contour(aes(z = prob), 
        color = "white", lty = 2, binwidth = br) +
    geom_point(
        data = (df %>% filter(win == 1)),
        color = "white", shape = 16, size = 2) +
    geom_point(
        data = (df %>% filter(win == 0)),
        color = "white", shape = 4, size = 2) +
    scale_fill_viridis(
        direction = 1, end = 0.85, option = "D", 
        breaks = seq(0, 1, br),
        labels = scales::percent_format()) + 
    coord_cartesian(xlim = c(10, 40), ylim = c(10, 40)) + 
    labs(title = "Super Dumb Super Bowl Model", 
        x = "Points For", y = "Points Against", fill = "Prob") + 
    theme(panel.background = element_rect(fill = "white"))

The behaviour of this model suddenly makes some sense (small data, and big upsets).

There’s nothing really new about this type of plot. It’s just a bunch of tile, contour and point layers and the viridis colourmap but I’ve been calling it the “Starry Night Plot” because it kind of looks like the painting. And I wanted to write about it because it has been super helpful in diagnosing and explaining model behaviour to other people.

I guess I hope you might be able to use it.

P.S. I also tried building models on point differentials for the exact matchups and conditioning the points for/against on Strength of Schedule… but then I realized that I was p-hacking…

P.S.S. Go Falcons.

Hypothetical Outcome Plots

Fri, 03 Feb 2017 00:00:00 -0000

suppressPackageStartupMessages(library(tidyverse))
# devtools::install_github("dgrtwo/gganimate")
# install.packages("cowplot")
library(gganimate)
# install image magick in terminal >> "brew install image magick"

If the Weatherman says that there is 30% chance of rain tomorrow. And it rains. Was he wrong? It’s an important question. Because it rained on November 8th. And a lot of people think that it wasn’t supposed to.

Communicating uncertainty is hard. It’s hard because uncertainty can be convoluted and opaque. And it’s hard because a lot of people can’t get down with boxplots or measures of spread.

Take this toy data for example:

set.seed(2016)

df <- tibble(
    `Blue Team` = round(rnorm(50, mean = 48, sd = 5)), 
    `Red Team` = `Blue Team` + round(rnorm(50, mean = -1, sd = 3))
    ) %>% 
    mutate(simulation = row_number()) %>% 
    gather(team, polling, -simulation) %>% 
    mutate(polling = ifelse(polling >= 100, 100, polling) / 100)

Imagine that this data was generated from some model. How might we represent the uncertainty in our model and around our predictions?

Perhaps we might push the data through a boxplot:

df %>% 
    ggplot(aes(x = team, y = polling)) +
    geom_boxplot()

Or a density chart:

df %>% 
    ggplot(aes(x = polling, fill = team)) + 
    geom_density(alpha = 1/2) + 
    scale_fill_manual(values = c("blue", "red"))

Or use some errorbars:

df %>% 
    group_by(team) %>% 
    summarise(
        mean = mean(polling), 
        low = quantile(polling, 0.025),
        high = quantile(polling, 0.975)) %>% 
    ggplot(aes(x = team, y = mean)) +
    geom_errorbar(aes(ymin = low, ymax = high))

These options all kind of suck. They’re not super intuitive. And they aren’t all that convincing, because they can intimidate a lot of people!

Instead of boxplots or density charts or regular errorbars we can hack errorbars to generate a proto-Hypothetical Outcome Plot:

df %>% 
    ggplot(aes(x = team, y = polling)) +
    geom_errorbar(aes(ymin = polling, ymax = polling))

Hypothetical Outcome Plots (HOPs) are a way to build and visualize uncertainty in the same way that we experience it (in and by countable events). The depth and theory behind HOPs is beyond the scope of this quick post, but if you’re interested in learning more check out this awesome Medium story by the UW Interactive Data Lab.

Extending our hacked errorbars with gganimate we can implement an actual HOP in just a few lines of R:

p <- df %>% 
    ggplot(aes(x = team, y = polling, frame = simulation)) +
    geom_errorbar(aes(ymin = polling, ymax = polling))

gganimate(p, title_frame = FALSE)

In this way we can directly experience the uncertainty of our model and the predictions that it happens to make.

In looking at the HOP it seems as though the Blue Team is supposed to win our imaginary game. Importantly, however, the Red Team comes up on top in certain simulations. So, don’t say that the model was wrong if they happen to actually win the imaginary game!

If you want to add a little polish to our HOP, I might suggest extending it with some ghost bars:

p <- df %>%
    ggplot(aes(x = team, y = polling)) +
    geom_errorbar(aes(
        ymin = polling, ymax = polling, 
        frame = simulation, cumulative = TRUE), 
        color = "grey80", alpha = 1/8) +
    geom_errorbar(aes(
        ymin = polling, ymax = polling, frame = simulation), 
        color = "#00a9e0") +
    scale_y_continuous(
        limits = c(0, 1), 
        labels = scales::percent_format()) +
    theme(panel.background = element_rect(fill = "#FFFFFF")) +
    labs(title = "", y = "Polling %", x = "")

gganimate(p, title_frame = FALSE)

But that’s it. Thanks for reading!

I have two more posts in the queue about visualizing models and model performance. Look for them in the near future!

Visualizing The Best

Sat, 21 Jan 2017 00:00:00 -0000

How do you measure “The Best”?

It’s not immediately clear. Because, “The Best” is incredibly vague and subjective. My “Best” is not the same as your “Best”. And our “Best”s can converge and diverge depending on what we are measuring and how we measure it.

I think “The Best” is often the wrong question. Usually when we’re looking for “The Best” (are you sick of me saying “The Best” yet?) we’re really just trying to find “The Better”.

Answers to questions like “Who is the best player in the NBA?” or “What is the best city in the world?” or “Which Pokemon is the best?” are fraught with caveats and asterisks and clarifications. And they have to be! What do you mean “Best”? Best Shooter? Best of All Time? Best Last Year? Best in terms of Quality of Living? Best on measures of Entertainment? Best Speeds? Best Attacks? Best Best Best Best Best! Aggh!

To answer these “Best” questions we have to narrow down the problem and convert them into “Better” questions. By slimming down the pool of possible options we can actually start to make some progress!

“Who is the better basketball player: Steph Curry, Kawhi Leonard, or Larry Bird?”
“Which city is better: Toronto, London, or San Francisco?”
“Which starter Pokemon (from Gen 1) is better for my team?”

These are questions, I think, we can actually answer! But only if we’re explicit about the measures we’re using in our calculations and the weights that we assign to them.

For instance, if we want to find the better basketball player we could come up with some formula that includes assists, shooting, usage, defense, and rebounds. Maybe my formula is y = 2 * Shooting + 1.2 * Usage - 3 * Defense + 1.1 * Assists^2 + 1.3 * Rebounds. But I think reducing five incredibly rich measurements down to one value is absurd. It’s lossy compression! And you might not agree with my formula. You could have a better one. You might really value rebounds. Or you might want to replace usage with steals! Blah Blah Blah Blah Blah.

Worry not! We’ve finally reached the part where I present a method for finding “The Best” (or “The Better”, at least). A visulization for “The Best”. It is inspired by (more like entirely ripped off from) this FiveThirtyEight article.

All that is required is the tidyverse and forcats

Basketball

Step 1: Spin up the data

df <- tribble(
    ~player, ~assists, ~shooting, ~usage, ~defense, ~rebounds,
    "Larry Bird", 88, 89, 93, 92, 87,
    "Kawhi Leonard", 71, 94, 92, 93, 62,
    "Stephen Curry", 95, 92, 87, 43, 32,
    "Top five average\n('97 to '15)", 72, 85, 32, 34, 30) %>% 
    gather(stat, percentile, -player) %>% 
    mutate(outof4 = percentile %/% 25 + 1) %>% 
    mutate(col = ifelse(
        player == "Top five average\n('97 to '15)", "A", "B")) %>% 
    mutate(order = recode(stat, 
        assists = 5, shooting = 1, usage = 2, 
        defense = 3, rebounds = 4)) %>% 
    mutate(stat = recode(stat, 
        assists = "ASSIST\nRATE", 
        shooting = "TRUE\nSHOOTING", 
        usage = "USAGE\nRATE", 
        defense = "DEFENSIVE\nBPM", 
        rebounds = "REBOUND\nRATE")) %>% 
    mutate(stat = factor(stat))

Step 2: Graph!

df %>% 
    ggplot(aes(x = fct_reorder(stat, order), y = outof4)) + 
    geom_col(alpha = 0.5, aes(fill = col), 
        width = 1, show.legend = FALSE, color = "white") +
    geom_hline(yintercept = seq(0, 4, by = 1), 
        colour = "#949494", size = 0.5, lty = 3) +
    geom_vline(xintercept = seq(0.5, 5.5, 1), 
        colour = "#949494", size = 0.4, lty = 1) +
    facet_wrap(~player) +
    coord_polar() + 
    scale_fill_manual(values = c("#e4e4e4", "#00a9e0")) + 
    scale_y_continuous(
        limits = c(0, 4), 
        breaks = c(1, 2, 3, 4)) + 
    labs(x = "", y = "") + 
    theme(
        panel.background = element_rect(fill = "#FFFFFF"),
        plot.background = element_rect(fill = "#FFFFFF"),
        strip.background = element_rect(fill = "#FFFFFF"),
        strip.text = element_text(size = 10),
        panel.grid = element_blank(),
        axis.ticks = element_blank(),
        axis.text.y = element_blank(),
        axis.text.x = element_text(size = 7),
        panel.spacing = grid::unit(2, "lines"))

I’m not sure what to call this new type of visualization, but I love it! I love it because you can immediately see that Larry Bird is “The Best” basketball player, at least when benchmarked against Steph and Kawhi. And I love it because it’s not just exotic fluff. The bubbles actually help with the interpretation of the data. The bubbles are simply better than columns, in this instance.

Just compare… These two graphs literally contain the same data:

df %>% 
    ggplot(aes(x = stat, y = percentile, fill = player)) + 
    geom_col(alpha = 0.5, color = "white")

df %>% 
    ggplot(aes(x = stat, y = percentile, fill = player)) + 
    geom_col(alpha = 0.5, color = "white",
        position = position_dodge())

Like, I can sort of tell that all the blue bars are really tall, but I can’t see much beyond that. It’s hard to jump back and forth between the stats and the players and tease out patterns.

Pokemon

Moving to the Pokemon question we can grab data from http://pokemondb.net/

df <- tribble(
    ~Pokemon, ~HP, ~Attack, ~Defense, ~SpAtt, ~SpDef, ~Speed,
    "Charizard", 78, 84, 78, 109, 85, 100,
    "Blastoise", 79, 83, 100, 85, 105, 78,
    "Venusaur", 80, 82, 83, 100, 100, 80) %>% 
    gather(stat, value, -Pokemon) %>% 
    mutate(outof6 = value %/% 20 + 1) %>% 
    mutate(order = recode(stat, 
        HP = 6, Attack = 1, SpAtt = 2, Defense = 3, SpDef = 4, Speed = 5))

Display it in the same way:

df %>% 
    ggplot(aes(x = fct_reorder(stat, order), 
        y = outof6, fill = Pokemon)) + 
    geom_col(alpha = 3/4, width = 1, 
        show.legend = FALSE, color = "white") +
    geom_hline(yintercept = seq(0, 6, by = 1), 
        colour = "#949494", size = 0.5, lty = 3) +
    geom_vline(xintercept = seq(0.5, 5.5, 1), 
        colour = "#949494", size = 0.4, lty = 1) +
    facet_wrap(~Pokemon) +
    coord_polar() + 
    scale_fill_manual(values = c("#06AED5","#ED933C", "#65B54F")) + 
    scale_y_continuous(
        limits = c(0, 6), 
        breaks = c(1, 2, 3, 4, 5, 6)) + 
    labs(x = "", y = "") + 
    theme(
        panel.background = element_rect(fill = "#FFFFFF"),
        plot.background = element_rect(fill = "#FFFFFF"),
        strip.background = element_rect(fill = "#FFFFFF"),
        strip.text = element_text(size = 10),
        panel.grid = element_blank(),
        axis.ticks = element_blank(),
        axis.text.y = element_blank(),
        axis.text.x = element_text(size = 7),
        panel.spacing = grid::unit(2, "lines"))

And we can see that there really is no objective “Best” this time. It totally depends on what measures are important to us. Perhaps, I really value Speed and Attack in my Pokemon. Well, looking at the graph I can see that I ought to grab Charizard. And you might decide to go with Blastoise because you like big tanky defense pokemon. It totally depends! And single point value would be incredibly misleading here.

Cities

Just one last example to wrap it all up. Using the PWC Cities of Opportunity Index I can grab the measures that are important me, like, Broadband Quality (need that fast internet!), Entertainment, Quality of Living, Ease of Starting a Business, and Cost of Living to generate similar comparisons.

PWC actually has a visulatization tool that spits out:

But I think my bubbles are better!

df <- tribble(
    ~city, ~Broadband, ~Entertainment, ~`QOL`, ~Startup, ~`COL`,
    "Toronto", 17, 16, 30, 30, 12, 
    "London", 19, 30, 16, 20, 1,
    "San Francisco", 20, 13, 18, 18, 6) %>%
    gather(stat, value, -city) %>% 
    mutate(score = value %/% 6 + 1) %>% 
    mutate(order = recode(
        stat, Broadband = 5, Entertainment = 1, 
        COL = 2, QOL = 3, Startup = 4))

df %>% 
    ggplot(aes(x = fct_reorder(stat, order), y = score, fill = city)) + 
    geom_col(alpha = 1, width = 1, show.legend = FALSE, color = "white") +
    geom_hline(yintercept = seq(0, 6, by = 1), 
        colour = "#949494", size = 0.5, lty = 3) +
    geom_vline(xintercept = seq(0.5, 5.5, 1), 
        colour = "#949494", size = 0.4, lty = 1) +
    facet_wrap(~city) +
    coord_polar() + 
    scale_fill_manual(values = c("#0C2238","#BF4E22", "#AF0023")) + 
    scale_y_continuous(
        limits = c(0, 6), 
        breaks = c(1, 2, 3, 4, 5, 6)) + 
    labs(x = "", y = "") + 
    theme(
        panel.background = element_rect(fill = "#FFFFFF"),
        plot.background = element_rect(fill = "#FFFFFF"),
        strip.background = element_rect(fill = "#FFFFFF"),
        strip.text = element_text(size = 10),
        panel.grid = element_blank(),
        axis.ticks = element_blank(),
        axis.text.y = element_blank(),
        axis.text.x = element_text(size = 7),
        panel.spacing = grid::unit(2, "lines"))

There you have it. “The Best” Visualization. Or at least a visualization for “The Best”.

Candlesticks Repurposed

Sun, 15 Jan 2017 00:00:00 -0000

My cousin Eli is way cooler than me. Well, to be fair, I don’t exactly set a high bar! Eli is a skydiving enthusiast with over 200 jumps to his name. And I read books and write nonsense like this…

Eli is meticulous about collecting metadata from each of his jumps. Just a glance at his logbook and you can tell there’s something interesting hiding in there. So. I asked him if I could play with his data and try to come up with something.

I don’t know the technical terms, but basically, when you think about it, there are, hopefully, three stages in every skydive. The jump, the pull, and the landing. I thought that I could do something interesting with those three events across each jump. But the histograms and line graphs and bar charts that I’ve come to rely on just weren’t working. Interesting data deserves an interesting visualization.

After some time messing around with graphical primitives in ggplot2 I hit on a candlestick-esque chart built from geom_segments that I thought could work.

This was my first attempt with simulated data:

After getting Eli’s logbook to conform to and look like the simulated data I threw it all back into the geom_segments and got this:

A little busy for my tastes. So, I decided to break every rule in data visualization and strip away all the fuss. I got rid of the labels and axes and text and titles and the legends because it was all noise and getting in the way of Eli’s fearlessness (stupidity?) and the data. I wanted the raw stuff to shine.

This is what I finally arrived at:

I love this slimmed down version because it’s kind of like putting notches in a wall. And despite the lack of labels and text annotations you can still see a lot. For instance, you can see Eli’s training exercises (two short bursts on the second and third row as well as the string of short jumps near the end of the third row). You can tease out some semblance of consistency in his free falls. And you can see where it almost all came to an end…that data point near the end of the second row where he had to pull his reserve chute.

Anyways. Thanks for letting my play with your data, Eli! For those interested, this is the ggplot2 code that I used to create the final graph:

df %>%
    ggplot(aes(x = id)) +
    geom_segment(size = 1, color = "grey20", aes(
        xend = id,
        y = jump,
        yend = chute)) +
    geom_segment(lty = 1, size = 0.25, color = "grey40", aes(
        xend = id,
        y = chute,
        yend = 0)) +
    facet_wrap(~section, scales = "free_x", ncol = 1) +
    labs(title = "", x = "", y = "", color = "") +
    scale_x_continuous(breaks = seq(0, 500, 25)) +
    theme(
        panel.background = element_rect(fill = "#FFFFFF"),
        panel.grid.major = element_blank(),
        panel.grid.minor = element_blank(),
        strip.text = element_blank(),
        axis.ticks = element_blank(),
        legend.position = "none",
        axis.text = element_blank())

Fantasy Hockey with rvest and purrr

Sun, 08 Jan 2017 00:00:00 -0000

rvest and purrr are wonderful bedfellows. The packages share the underlying tidyverse API. And it feels simple and almost natural to combine them when scraping the web.

Here is a slimmed down and worked recipe of how to leverage rvest and purrr in Fantasy Hockey.

Step 0. Load packages.

library(tidyverse)
library(rvest)
library(purrr)
library(stringr)

Step 1. Find a data source.

I’m going to use Fantasy Sports Portal for this example.

Step 2. Figure out the CSS selectors for the data.

SelectorGadget makes this dead simple, like so:

Step 3. Fetch the data elements.

I like to put everything in tibble as soon as possible and use stringr to adjust the url for the different position pages. You’ll notice that I’m only grabbing name and goals. Feel free to grab whatever!

p_fetch <- function(position = "C") {

    url <- str_c(sep = "",
        "https://www.fantasysp.com/projections/hockey/weekly/",
        position)

    page <- read_html(url)

    names <- page %>%
        html_nodes("td:nth-child(2)") %>%
        html_text()

    goals <- page %>%
        html_nodes("td:nth-child(4)") %>%
        html_text()

    df <- tibble(name = names, goals)

    return(df)
}

Step 4. Iterate through each page.

Instead of writing a for loop, I like to use pmap from purrr to iterate through the Centre, Left-Wing, Right-Wing and Defense position projection pages (I left out the Goalies for obvious reasons).

p_pull <- function() {

    params <- tibble(position = c("C", "LW", "RW", "D"))

    df <- params %>%
        pmap(p_fetch) %>%
        bind_rows()

    return(df)
}

Step 5. Clean and format the projection data.

This is a pretty janky use separate but it works to get everything into a format that I like.

p_clean <- function() {

    df <- p_pull() %>%
        separate(name,
            into = c("junk", "first", "last", "meta"),
            sep = "(?=[A-Z][a-z])|(?<=[a-z])(?=[A-Z])",
            fill = "right",
            extra = "merge") %>%
        separate(meta, into = c("team", "position"), sep = "\\s") %>%
        mutate(name = str_c(first, last, sep = "")) %>%
        mutate(goals = as.numeric(goals)) %>%
        drop_na() %>%
        mutate(length = str_length(team)) %>%
        filter(length <= 3) %>%
        select(name, team, position, goals)

    return(df)
}

df <- p_clean()

At this point you should have a nice clean tibble/dataframe with every player, their position, team, and their projected goals for this week. I could stop here, but I wanted to go a little further with a value over replacement player (VORP) calculation.

Step 6. Calculate a replacement player for each scoring position.

I’m using pmap again to pump through each position to get the mean value for the top X players. It’s a little overkill, but really flexible.

p_replacement <- function(pos, slots) {

    rp <- df %>%
        filter(position == pos) %>%
        arrange(desc(goals)) %>%
        filter(row_number() <= slots) %>%
        group_by(position) %>%
        summarise(goals = mean(goals))

    return(rp)
}

p_vorp <- function() {

    # slots depend on how many position players start for each team
    # if there are 10 teams and 2 LW per team then slots -> 10 * 2 = 20

    params <- tribble(
        ~pos, ~slots,
        "C", 20,
        "LW", 20,
        "RW", 20,
        "D", 20)

    rp <- params %>%
        pmap(p_replacement) %>%
        bind_rows()

    return(rp)
}

Step 7. Calculate the VORP for each player.

Simple join at this point…

replacement <- p_vorp()

# calculate value over replacement player

vorp <- df %>%
    left_join(replacement, by = "position") %>%
    mutate(goals_vorp = goals.x - goals.y) %>%
    rename(goals = goals.x, goals_rp = goals.y) %>%
    select(-goals_rp) %>%
    arrange(desc(goals_vorp))

And that’s it!

Sleep Circle

Sat, 31 Dec 2016 00:00:00 -0000

I hate pie charts. They’re incredibly misleading and way overused. Walter Hickey has a great post on why you should never use them, but the entire argument can basically be summed up in an image:

Last night, while browsing /r/dataisbeautiful, however, I spied a pie chart that I actually kind of liked. I’m talking about this thing:

My daughters sleeping patterns for the first 4 months of her life. One continuous spiral starting on the inside when she was born, each revolution representing a single day is sitting at 52.4k upvotes and is, right now, the most liked post on /r/dataisbeautiful. Ever.

It seems the pie chart format is well suited for sleep data because time spirals like a clock. It doesn’t abruptly stop and start again on the other side of a bar or column. The graph is powerful, this once, because it helps to show circadian rhythms (and an infant finding hers).

I thought it would be fun to recreate /u/andrew_elliott’s graph, but the process he detailed in the comments was fairly opaque and unhelpful:

“Visualisation pattern was created using the CAD package Rhinoceros with Grasshopper plugin, using my own script. Then Adobe Illustrator was used for appearance. Source data was manually logged using Baby Connect iPhone app by my wife and myself.”

After futzing around in R for a couple of hours, with data that I simulated, I think I’ve come close to a viable recreation.

Here’s my version of /u/andrew_elliott’s Sleep Circle / Time Spiral Graph:

It’s obviously not an exact copy (it couldn’t be with fake data). But I think it captures everything I liked about the original.

If you’re interested in creating sleep-cirle-time-spiral graphy type things for yourself, perhaps with your own FitBit data (?), the script that underpins this post is available on GitHub

Kindle Clippings

Fri, 23 Dec 2016 00:00:00 -0000

I highlight a lot of junk on my Kindle. Well, it’s not all junk! 💩 There’s usually some good stuff buried deep within my clippings.txt file. But it’s hard to manually parse through the file (and the junk).

In the past I’ve relied on online tools to organize my clippings. But this year, I thought I’d try and build my script to manage and clean it all.

Here’s what I ended up with:

library(tidyverse)
library(stringr)
txt <- read_lines("clippings_archive_2016.txt")

parse_clippings <- function(txt) {

    h1_tibble <- txt %>% as_tibble()

    h2_prepend <- bind_rows(tibble(value = "=========="), h1_tibble)

    h3_index <- h2_prepend %>%
        mutate(index = ifelse(
            str_detect(pattern = "==========", value), 0, NA)) %>%
        mutate(row = row_number()) %>%
        mutate(row = ifelse(index == 0, row, NA)) %>%
        fill(row, .direction = "down") %>%
        mutate(entry = row %/% 5 + 1) %>%
        group_by(entry) %>%
        mutate(meta = row_number())

    meta <- tribble(
        ~meta, ~type,
            1, "start",
            2, "book",
            3, "location",
            4, "blank",
            5, "highlight"
    )

    h4_meta <- h3_index %>%
        left_join(meta, "meta") %>%
        select(entry, value, type) %>%
        spread(type, value) %>%
        ungroup()

    h5_separate <- h4_meta %>%
        select(book, location, highlight) %>%
        separate(location,
            into = c("page", "location", "date"),
            sep = "\\s\\|\\s", fill = "right") %>%
        separate(book,
            into = c("book", "author"),
            sep = "\\s\\(", extra = "merge", fill = "right")

    h6_replace <- h5_separate %>%
        mutate(author = str_replace_all(
            author, pattern = "\\)", "")) %>%
        mutate(page = str_replace_all(
            page, "\\-\\sYour\\sHighlight\\son\\spage\\s", "")) %>%
        mutate(location = str_replace_all(
            location, "Location\\s", "")) %>%
        mutate(date = str_replace_all(
            date, "Added\\son\\s", ""))

    h7_format <- h6_replace %>%
        drop_na() %>%
        mutate(date = as.Date(date,
            format = "%A, %B %d, %Y %I:%M:%S %p")) %>%
        mutate(page = as.integer(page))

    return(h7_format)
}

clippings <- parse_clippings(txt)

I’ve just pumped my clippings from 2016 through the script and everything seems to be working! I’m sifting through each highlight right now, trying to curate them down my to annual “Favourite Quotes of YYYY” post. Should have it up tomorrow!

Goodreads API

Fri, 23 Dec 2016 00:00:00 -0000

It’s December 23rd and I’ve only read 49 books. Whoops. There’s still time, but it’s definitely getting dicey. I’m about halfway through three books right now so I think I’ll be able to pull it off. Fingers crossed.

Of course, last year I did 52 books in 52 weeks and remember sitting pretty just before Christmas.

As I’ve been logging all my activity on Goodreads I thought it would be neat to plug into the API and compare my reading progress between the years. To see if I read at the same pace and whether there’s some sort of seasonality in my reading habits.

If you happen to use Goodreads and want to do the same here’s how I did it:

Setup

(Man, I love Hadley)

# load packages
library(httr)
library(tidyverse)
library(stringr)
library(xml2)
library(viridis)
library(knitr)

# knitr options
opts_chunk$set(cache = TRUE, warning = FALSE, message = FALSE)

API Guide

I’ve censored my API_KEY and GR_ID but if you replace the "XXXXXXXXXXXXX"s with your KEY and your ID you should be good to go!

# API_KEY <- "XXXXXXXXXXXXX"
# GR_ID   <- "XXXXXXXXXXXXX"

URL <- "https://www.goodreads.com/review/list?"

Get Shelf

This is where the heavy lifting GETs done. I’m leaning on httr and XML2 to parse the API responses.

get_shelf <- function(GR_ID) {
    shelf <- GET(URL, query = list(
        v = 2, key = API_KEY, id = GR_ID, shelf = "read", per_page = 200))
    shelf_contents <- content(shelf, as = "parsed")
    return(shelf_contents)
}

shelf <- get_shelf(GR_ID)

get_df <- function(shelf) {

    title <- shelf %>%
        xml_find_all("//title") %>%
        xml_text()

    rating <- shelf %>%
        xml_find_all("//rating") %>%
        xml_text()

    added <- shelf %>%
        xml_find_all("//date_added") %>%
        xml_text()

    started <- shelf %>%
        xml_find_all("//started_at") %>%
        xml_text()

    read <- shelf %>%
        xml_find_all("//read_at") %>%
        xml_text()

    df <- tibble(
        title, rating, added, started, read)

    return(df)
}

df <- get_df(shelf)

Clean

After getting the XML data into my IDE I tabled and cleaned the data with dplyr and tidyr.

get_books <- function(df) {

    books <- df %>%
        gather(date_type, date, -title, -rating) %>%
        separate(date,
            into = c("weekday", "month", "day", "time", "zone", "year"),
            sep = "\\s", fill = "right") %>%
        mutate(date = str_c(year, "-", month, "-", day)) %>%
        select(title, rating, date_type, date) %>%
        mutate(date = as.Date(date, format = "%Y-%b-%d")) %>%
        spread(date_type, date) %>%
        mutate(title = str_replace(title, "\\:.*$|\\(.*$|\\-.*$", "")) %>%
        mutate(started = ifelse(
            is.na(started), as.character(added), as.character(started))) %>%
        mutate(started = as.Date(started)) %>%
        mutate(rating = as.integer(rating))

    return(books)
}

books <- get_books(df)

Compare

All of that get to this graph:

It’s funny to see that I started strong in both years and fell off sometime around March. Though I recovered somewhat in 2015, Spring 2016 was a bad season for reading, apparently.

Looks like I was finished 52 books by December 21st last year. Whoops. Oh well, I still think I can mad rush it to the finish line.

Stratford

Sat, 17 Dec 2016 00:00:00 -0000

I started going to the Stratford Festival with my grandparents when I turned seven. Honestly, I hated that first time. Shakespeare and seven-years-old go together about as well as peanut butter and snot. But, it meant a lot to my grandparents. So, I did the Festival again when I was eight.

I hated that second time, too. But the third time we did the Festival I hated it a little bit less. And the fourth time I actually kind of enjoyed it.

By the eighth grade Shakespeare had become a significant enough part of my life that I decided to write my speech on The Bard. I can’t remeber it all exactly but it was something like, “Why you should give Shakespeare a chance in High School” (I was really popular…)

Now, Shakespeare at the Stratford Festival has become an incredibly rich tradition for me and my grandparents. We’ve done the festival for 16 years in a row! And we’re just now starting to plan our seventeenth adventure. The only problem: We’re running out of plays to see! And it’s getting harder and harder to remember what we’ve actually seen.

This year, Stratford is running Twelfth Night, Romeo and Juliet and Timon of Athens. I was sure we had seen Twelfth Night and Romeo and Juliet so our decision was pretty easy. But still, I was kind of curious about what was still left…

Data Investigation

Simply pulling and graphing the runs for every play for single season didn’t really get me anywhere…

Trying to get fancy with lines instead of points didn’t help either…

Zooming in the Histories was interesting and telling. Notice how the Henry plays are bunched up and played at the same time during the same season…

But zooming back out and colouring all the genres on the same vertical line plot really didn’t work…

Faceting the genres with three columns was a disaster…

And a one column facet wasn’t much better…

I was starting to get somewhere when I reverted back to points (instead of lines) and when I reordered the data accoring to the most recent runs of each play…

Moving the y-axis text over…

And trimming the data to 2000 helped immensely…

At this point, I was able to reverse out exactly the play that me and my grandparents had seen and when.

After trying to clean up the styling…

And trying to mess with the colors (gross)…

I finally…

…arrived at this polished and finished version.

Here, I present every play that me and my grandparents have seen so far (the Xs)…

The cool thing about this graphic is that you can infer what Stratford might play in the seasons to come (seems Coriolanus and The Two Noble Kinsmen are due for a run)…

If you’re interested in the code for each graph and the process I used to pull the data from Wikipedia it’s available on GitHub

HSX Arbitrage

Sun, 11 Dec 2016 00:00:00 -0000

TL;DR: I wrote a web scraper to help me make fake money.

I’ve been playing around with an online marketplace called the “Hollywood Stock Exchange” for a couple of months now. The website is basically “fantasy” for actors, directors and upcoming movies but it functions pretty much exactly like a stock market.

In simple terms here’s how the exchange works:

Rogue One (SW16) is currently trading at ~$440 on the HSX. This means that the market thinks that Rogue One will gross $440,000,000 during the first four weeks in theatres. For context, the last Star Wars movie (STAR7) did $936 Million in the same time period. If you think that Rogue One is going to do something in between $440 and $936 Million than you ought to buy the security because it will converge and delist on the actual box office totals.

There’s definitely money* to be made on a Rogue One position, right now. But, I think there’s an arbitrage opportunity in buying the actors and actresses attached to Rogue One instead.

See, actors and actresses, called StarBonds on the HSXm are priced slightly differently. StarBonds are derived from the average total box-office performance for the last five credited films by release date. This means that each time a movie featuring a particular star cashes out and delists, the box office gross is added into the star’s Trailing Average Gross (TAG), and the bond price is adjusted to match.

Mads Mikkelsen (MMIKK) a Rogue One cast member, for instance, is currently priced at $47.10. His actual TAG value, however, is only $43.54. This spread is because of Rogue One. In a couple weeks Rogue One will be added to Mads’ TAG value while at the same time some random indie art film called “A Royal Affair” (ROYAF) will fall off and out of his TAG.

I mean, I could manually do the math for the Mads example and figure out what his TAG will be when Rogue One gets added. But there’s no fun in that! And besides, Mads is already a big deal. The thing about the new Star Wars movies is that a bunch of nobodies get cast and turned it into Super Stars overnight. It means that a lot of random indie art films (think sub ~$5) are about to fall off to make room for Rogue One (~$440). Basically, Rogue One is going to make the TAGs for a lot of people pop.

The punch line:

I’m lazy. And, I didn’t want to manually do the math. So, I built a web scraper to calculate the arbitrage opportunities for me.

(If you don’t care about the code just scroll to the bottom…)

Setup

# load packages
library(tidyverse)
library(rvest)
library(purrr)
library(stringr)
library(knitr)

# knitr options
opts_chunk$set(cache = TRUE, warning = FALSE, message = FALSE)

# set base URL for HSX.com
URL <- "http://www.hsx.com/security/view/"

Cast Scraper

get_cast <- function(movie) {

    page <- read_html(str_c(URL, movie))

    name <- page %>% 
        html_nodes(".credit p") %>% 
        html_text()

    df <- tibble(name) %>% 
        separate(name, into = c("name", "symbol"), sep = "\\s\\(") %>%
        mutate(symbol = str_replace_all(symbol, "\\)", ""))

    return(df)
}

cast <- get_cast("SW16")

Credits Scraper

get_credits <- function(actor) {

    page <- read_html(str_c(URL, actor))

    movie <- page %>% 
        html_nodes(".credit span") %>% 
        html_text()

    date <- page %>% 
        html_nodes("strong") %>% 
        html_text()

    l <- length(movie)
    date <- date[1:l]

    df <- tibble(date, movie) %>% 
        mutate(symbol = actor)

    return(df)
}

credits <- map(cast$symbol, get_credits) %>% bind_rows()

Clean Credits

clean_credits <- function(credits) {

    clean <- credits %>% 
        mutate(date = as.Date(date, format = "%b %d, %Y")) %>% 
        mutate(date = ifelse(!is.na(date), 
            as.character(date), as.character("3000-01-01"))) %>% 
        mutate(date = as.Date(date)) %>% 
        group_by(movie, symbol) %>% 
        mutate(future = ifelse(date >= Sys.Date() | is.na(date), TRUE, FALSE))

    five <- clean %>% 
        filter(future != TRUE) %>% 
        arrange(symbol, desc(date)) %>% 
        group_by(symbol) %>% 
        mutate(movie_idx = row_number()) %>% 
        filter(movie_idx <= 5) %>% 
        select(-future) %>% 
        ungroup()

    return(five)
}

tag_credits <- clean_credits(credits)

Brute-force Search

movies <- tag_credits %>% 
    distinct(movie, .keep_all = FALSE) %>% 
    mutate(search_term = str_replace_all(movie, "\\s", "\\+"))

get_meta_m <- function(movie) {

    page <- read_html(str_c(
        "http://www.hsx.com/search/?keyword=", 
        movie, 
        "&status=ALL&action=submit_advanced"))

    movie <- page %>% 
        html_nodes("td:nth-child(1)") %>% 
        html_text()

    symbol <- page %>% 
        html_nodes("td:nth-child(2)") %>% 
        html_text()

    price <- page %>% 
        html_nodes("td:nth-child(5)") %>% 
        html_text()

    l <- length(price)
    movie <- movie[1:l]
    symbol <- symbol[1:l]

    df <- tibble(movie, symbol, price)

    return(df)
}

meta_m <- map(movies$search_term, safely(get_meta_m))

Clean Search Results

clean_meta_m <- function(meta_m) {

    meta_t <- meta_m %>% transpose()
    is_ok <- meta_t$error %>% map_lgl(is_null)
    meta_k <- meta_t$result[is_ok]

    df <- meta_k %>% 
        bind_rows() %>% 
        distinct(.keep_all = TRUE) %>% 
        filter(!str_detect(movie, "H\\$")) %>% 
        filter(!str_detect(symbol,"\\."))

    return(df)
}

meta_m <- clean_meta_m(meta_m)

Join Tickers and Prices

tag_prices <- tag_credits %>% 
    left_join(meta_m, by = "movie") %>% 
    mutate(date = as.Date(date)) %>% 
    mutate(price = as.numeric(str_replace(price, "H\\$", ""))) %>% 
    group_by(movie, symbol.x) %>% 
    slice(which.max(price)) %>% 
    select(date, symbol = symbol.x, movie, price, movie_idx) %>% 
    # correct minor mistakes for DYEN
    bind_rows(tribble(
        ~date, ~symbol, ~movie, ~price, ~movie_idx, 
        NA, "DYEN", "Legend of the Fist", 0.05, 4,
        NA, "DYEN", "Ip Man 2", 0.20, 5))

Cast Metadata Scraper

get_meta_a <- function(actor) {

    page <- read_html(str_c(URL, actor))

    meta <- page %>%
        html_nodes(".data_column td") %>%
        html_text()

    meta_split <- split(meta, ceiling(seq_along(meta)/2))

    meta_df <- tibble(meta = meta_split) %>% 
        mutate(meta = as.character(meta)) %>% 
        mutate(meta = str_replace_all(meta, '\\"|\\,|c\\(|\\)', "")) %>% 
        separate(meta, into = c("meta", "data"), sep = "\\:")

    value <- page %>% 
        html_nodes(".value") %>% 
        html_text()

    df <- meta_df %>% 
        spread(meta, data) %>%
        mutate(trading_tag = value) %>% 
        mutate(TAG = as.numeric(str_replace(TAG, "\\$", ""))) %>% 
        separate(trading_tag, into = c("junk", "trading_tag"), 
            sep = "H\\$", extra = "drop") %>% 
        mutate(trading_tag = as.numeric(trading_tag)) %>% 
        mutate(actual_tag = round(as.numeric(TAG)/1e6,2)) %>% 
        select(symbol = Symbol, trading_tag, actual_tag) %>% 
        mutate(symbol = str_replace_all(symbol, "\\s", ""))

    return(df)
}

meta_a <- map(cast$symbol, get_meta_a) %>% bind_rows()

TAG Forecasts

tag_new <- tag_prices %>% 
    filter(!(movie_idx == 5)) %>% 
    group_by(symbol) %>% 
    summarise(total = sum(price) + 440) %>% 
    mutate(tag_forecast = total / 5) %>% 
    select(-total)

Calculate Arbitrage Opportunities

arbitrage <- meta_a %>% 
    left_join(tag_new, by = "symbol") %>% 
    mutate(investment = 20000 * trading_tag) %>% 
    mutate(return = ifelse(tag_forecast >= trading_tag, 
        (20000 * tag_forecast), (20000 * tag_forecast * -1))) %>% 
    mutate(roi = return / investment)

Punch Line

Ha! I knew it. Arbitrage Galore. Just look at Mads. He’s trading at $43.54 right now. But when Rogue One gets added he’s going to pop to $129.28. But it’s not just Mads. The entire top billed cast of Rogue One is chronincally undervalued right now, with Donnie Yen (DYEN) at the extreme end. Basically an investment in DYEN could yield a return of ~5.5X. Damn, I really wish this wasn’t just fake money…