Joel Grus

2025 Year in Review

2025-12-22T12:00:00+01:00

Year In Review 2025

Vibe Coding 9 -- AI Agents That Play Taboo

2025-09-08T12:00:00+02:00

One thing that I've gotten pretty interested in recently is "agent engineering" or "agentic architectures" or "agentic system design" (I'm not sure there's a canonical name for it). Of course there is a lot of interesting work on "how do I use an LLM to solve some contained inference or generation problem."

But there's also some pretty interesting (to me) problems related to "how do I architect a system of autonomous agents toward some end?" What are the patterns for communication, and for orchestration, and for self-organization, and so on?

Every framework takes some point of view on these, but as always I am partial to dspy, which (as I understand it) is architecturally pretty unopinionated beyond "use signatures" and "use modules", and in particular about multi-agent coordination. Which is fine, I like the flexibility and challenge of figuring out these designs myself.

Already I've been building a lot of "agentic workflows," but one thing that I hadn't built yet is a real "system of agents" where they interact with each other and do things autonomously (as opposed to moving themselves through pre-determined workflows).

So I started thinking about what might be a fun, impractical example, and I came up with the idea of Taboo:

Taboo

If you're not familiar, it's a word-guessing party game, where there's a word to guess (e.g. "APPLE") and several taboo words (e.g. "RED", "FRUIT", "PIE", "COMPUTER", "MACINTOSH") that you're not allowed to use in your clues. At a party there would typically be two teams, and they would take turns: one team would "clue" and "guess" while the other monitors them and sounds a buzzer if they say any of the taboo words.

If you squint hard enough, this looks like a bunch of "agents":

Cluer (generate clues)
Buzzer (buzz if any of the clues are taboo)
Guesser (try to guess the word)
Judge (say if any of the guessers are correct)

What's more, this is not a turn-based game but more of a free-for-all: the cluer keeps shouting out clues while the guessers keep shouting out guesses while the buzzer and judge monitor for winning or losing.

This makes it somewhat interesting as a system of agents; each operates at its own pace rather than in a predetermined sequence.

ChatGPT and I game-planned this out and came up with a pretty complicated design involving multiple message queues and so on.

(Originally, I wanted to make it a collaborative web game where one or more humans could play as cluer or guesser along with the AI, but eventually I simplified it down to just letting agents play amongst themselves.) (Maybe I'll build the web version in the future.)

As usually happens on these projects, I started with ChatGPT's PRD, found the resulting code immensely overengineered, and spent quite a while stripping it down and simplifying it. It's still complicated because of all the async pieces, but I think it's a relatively clean design for what it does.

Design

At its core, the game is a stream of (typed) messages. A menagerie of agents listens to the stream, and each of them can at any time send its own message to the stream.

So, for example, the guessing agent (or human) sends messages that look like

class GuessEvent(BaseEvent):
    role: Literal["guesser"]
    player_id: str
    guess: str
    rationale: Optional[str] = None

and similarly for the other players.

There is a Player base class with a fair amount of shared plumbing in it, and then subclasses for each of the roles:

class Cluer(Player[ClueEvent], ABC):
    """
    Player that gives clues to help guessers guess the target word.
    """
    def __init__(self):
        super().__init__()

    async def next_clue(self) -> str:
        raise NotImplementedError

    async def play(self):
        while not self.game.is_over():
            clue = await self.next_clue()
            await self.announce(ClueEvent(role="cluer", clue=clue))

Most of the magic is in next_clue which is not implemented in the base class, but you can also see the play loop, which just continuously generates and announces clues until the game is over. (self.game has a history of gameplay so that your implementation of next_clue can take into account previous clues and guesses).

There is a Game class that sets everything in motion (details elided):

    async def play(self) -> Dict[str, Any]:
        async def timeout():
            await asyncio.sleep(self.duration_sec)
            await self.publish(SystemMessage(role="system", event="timeout"))

        # Launch players and timeout tasks
        player_tasks: list[asyncio.Task] = [
            asyncio.create_task(p.play()) 
            for p in self.players
        ]
        timeout_task = asyncio.create_task(timeout())

        try:
            while True:
                # wait for the next batch of events
                # iterate over each one
                # - if it's a timeout event, game over, lose
                # - if it's a buzzer event, game over, lose
                # - if it's a judge correct event, game over, win
                ...
        catch GameOverException:
            # stop all the agent work
            ...

(I feel like I am starting to get the hang of Python async.)

So now where does the AI come into things?

AI

The only dataset of Taboo cards I could find was in German, which was not particularly useful for my purposes. So I figured I might as well let the AI generate them. Initially I had the AI generate both the target word and the taboo words, but it kept picking "VOLCANO" all the time, so eventually I changed it so that you give it the target word and it comes up with the taboo words:

class CreateTabooWords(dspy.Signature):
    """
    You are creating a game card for the game Taboo. Each card has a target word
    and a list of taboo words. One of the players ("the cluer") will try to get
    the other players to guess the target word by giving clues, but they cannot
    use any of the taboo words in their clues, or they lose.

    For example, if the target word is "apple", the taboo words might be
    ["fruit", "red", "pie", "tree", "juice"].

    You will be given a target word. Your task is to come up with a list of 5 taboo words
    that will make the game interesting and challenging. The taboo words should ideally
    be the most obvious clues to the target word, so that the cluer has to be creative.
    """
    target: str = dspy.InputField(description="The target word for the game")
    taboo_words: list[str] = dspy.OutputField(
        description="A list of taboo words that cannot be used in clues for the target word")

create_taboo_words = dspy.ChainOfThought(CreateTabooWords)

A few things here: first, you can see that I gave this signature a more substantial docstring than I often do, because in this case there is quite a bit to its task that you can't just get from the input field and output field. (Ideally I would probably use few-shot Examples instead of putting it in the docstring, but oh well.)

But at the same time you can see that the docstring prompt is a pure description of the task. I don't try to cram in there what the input types or what the output types are. That's all still handled by the Signature machinery, so the docstring "feels like" a docstring.

I also want it to be very thoughtful about generating the taboo words, so I used dspy.ChainOfThought and (although you can't see it here) I set up this agent to use gemini-2.5-pro.

(In general for this project I had different agents use different models, depending on how hard / creative their tasks were.)

And then e.g. the guesser gets a signature that looks like

class GuessWord(dspy.Signature):
    """
    You are playing a game of Taboo. Your goal is to guess the target word based on the clues given by the Cluer.
    """

    history: list = dspy.InputField(description="The history of the game so far, including previous clues, buzzes, guesses, and judgments")
    player_id: str = dspy.InputField(description="The ID of the player making the guess")
    player_personality: str | None = dspy.InputField(description="Optional personality or background information about the player making the guess")

    guess: str = dspy.OutputField(description="The guessed word")
    rationale: str | None = dspy.OutputField(description="Optional rationale for the guess")

and has a next_guess function that just grabs the history from the game and the player information from itself and makes an async dspy call. (There is a little more ceremony around it to make the call cancelable, but that part is not super important here.)

One other problem I ran into is that the "buzzer" and "judge" were not doing a very good job (possibly because I gave them 2.5-flash-lite to use so they would be fast); they would e.g. declare some guess a winner when it wasn't.

Eventually I added an extra "justification" field to the signature and that seemed to keep them more on the straight and narrow:

class BuzzClue(dspy.Signature):
    """
    Check if the clue is one of the taboo words (or a minor variation, like singular/plural).
    It's ok if the clue has a similar meaning to the taboo word, it just can't be the same word.
    """
    clue: str = dspy.InputField(description="The clue word or phrase given by the Cluer")
    taboo_words: list[str] = dspy.InputField(description="The taboo words that cannot be used in the clue, or you lose")

    buzz: bool = dspy.OutputField(description="Whether the clue violates the taboo words or not")
    justification: str = dspy.OutputField()

I guess that's kind of a poor man's chain-of-thought.

And it all works. The whole thing is up on GitHub, as usual, so check it out:

https://github.com/joelgrus/taboollm

Here's one where it gets it really quick:

$ uv run python -m taboo play --target vibes
Card: target=vibes, taboo_words=['Feeling', 'Energy', 'Mood', 'Atmosphere', 'Vibrations']
[cluer] Aura

[guesser p2-friendly] Vibe — The clue "Aura" directly brings to mind "vibe" as a synonym for a distinctive atmosphere or feeling around a person or place.

[cluer] Spirit

[judge] Vibe by p2-friendly -> CORRECT

[system] end: correct, winner: p2-friendly

Round finished. Winner: p2-friendly

And here's a hard one:

$ uv run python -m taboo play --target convolution

Card: target=convolution, taboo_words=['Kernel', 'Filter', 'Image', 'Signal', 'Neural']

[cluer] Combining

[guesser p2-friendly] Mix — The clue "Combining" directly suggests the action of mixing things together.

[judge] Mix by p2-friendly -> INCORRECT

[guesser p3-sarcastic] Merging — "Combining." Wow, such a deep and insightful clue. It's almost as if things, when combined, become... *merged*. Who would've thought?

[guesser p2-friendly] Merge — "Combining" can mean bringing two or more things together to form a single entity, which is what "merge" implies, like companies merging.

[cluer] Folding

[judge] Merging by p3-sarcastic -> INCORRECT

[judge] Merge by p2-friendly -> INCORRECT

[guesser p3-sarcastic] Uniting — Since "merging" and "mixing" weren't quite abstract enough for the intellectual heights of "combining," perhaps we're going for something more profound. Like... bringing things together in a grand, unified front.

[judge] Uniting by p3-sarcastic -> INCORRECT

[guesser p2-friendly] Uniting — The clue is "Combining," and since "Mix" and "Merge" have been ruled out, I'm thinking of another way to describe bringing things together, especially in a more collective or harmonious sense. "Uniting" fits this idea well.

[judge] Uniting by p2-friendly -> INCORRECT

[guesser p1-thoughtful] MIX — The clue "Combining" directly describes the action of mixing multiple elements together. It's a very common and straightforward synonym.

[judge] MIX by p1-thoughtful -> INCORRECT

[cluer] Slide

[guesser p2-friendly] Kneading — The first clue was "Combining," and then the second clue was "Folding." When you combine ingredients and then fold them, especially in cooking, it often refers to the process of kneading dough. This combines both actions.

[judge] Kneading by p2-friendly -> INCORRECT

[guesser p3-sarcastic] Consolidate — Ah, yes, 'Combining,' followed by the subtle hint of 'Folding.' Clearly, the grand master is guiding us towards the profound concept of... consolidating. Because when you combine and fold, you obviously... consolidate. The intellectual journey is truly breathtaking.

[judge] Consolidate by p3-sarcastic -> INCORRECT

[cluer] Overlap

[guesser p1-thoughtful] Incorporate — The first clue "Combining" points to bringing things together. The second clue "Folding" suggests a specific way of combining, often by bringing something into an existing structure or mixture. "Incorporate" means to take in or include something as part of a whole, which directly relates to combining, and it also fits the idea of "folding" something into a plan, system, or mixture.

[judge] Incorporate by p1-thoughtful -> INCORRECT

[guesser p1-thoughtful] Tuck — The clues seem to build on ways of arranging or inserting things. "Combining" is a general action. "Folding" is a specific way of preparing something to be combined or inserted. "Slide" suggests a smooth, deliberate movement, often into a confined space. "Overlap" implies that one part rests on or covers another. "Tuck" fits well with all these: you combine things by tucking one into another; you often fold something before you tuck it; you slide something into place when tucking it; and the tucked element will often overlap with the material it's tucked into.

[judge] Tuck by p1-thoughtful -> INCORRECT

[guesser p2-friendly] Joint — The clue \"Combining\" suggests bringing different elements together. \"Folding\" implies a bend or crease, which is characteristic of certain types of connections. \"Slide\" refers to a smooth movement of one part past another. A \"joint\" serves as a point where things are combined, and many types of joints facilitate either a folding motion (like a hinge) or a sliding motion. It seems to connect all three clues well.

[judge] Joint by p2-friendly -> INCORRECT

[cluer] Aggregation

[guesser p1-thoughtful] Weave — The clues "Combining," "Folding," "Slide," and "Overlap" strongly suggest the process of weaving. Weaving involves combining threads, which are folded or passed over and under each other. The threads slide past each other, and they naturally overlap to form a fabric. This integrates all the clues into a single coherent action.

[judge] Weave by p1-thoughtful -> INCORRECT

[guesser p2-friendly] Weave — The clues "Combining," "Folding," "Slide," and "Overlap" all strongly point to the action of weaving. When you weave, you are combining threads or strands. The process involves threads folding over and under each other, sliding past each other, and continuously overlapping to create fabric or a pattern. It perfectly encapsulates all the actions described.

[judge] Weave by p2-friendly -> INCORRECT

[cluer] Sweep

[guesser p3-sarcastic] Collapse — Ah, the intellectual odyssey continues! We began with the majestic 'Combining,' then progressed to the intricate 'Folding,' and now, the profound 'Slide.' It's almost as if things, when combined and folded, have a tendency to, you know, just completely give up and slide into a heap. The sheer elegance of these clues points directly to something... collapsing. It's truly a masterclass in subtlety.

[judge] Collapse by p3-sarcastic -> INCORRECT

[guesser p2-friendly] Origami — The clue "Combining" is a general action, reinforced by "Aggregation." "Folding" is a very specific action, which, along with "Slide" and "Overlap," strongly suggests a process of manipulating material. Origami perfectly encompasses all these: it involves extensive folding of paper, parts of the paper slide against each other during the folding process, layers of paper constantly overlap, and the entire creation is a combination or aggregation of these folds. It ties all five clues together cohesively.

[judge] Origami by p2-friendly -> INCORRECT

[cluer] Superimpose

[guesser p1-thoughtful] Bundle — The clues provided are "Combining," "Folding," "Slide," "Overlap," and "Aggregation." "Bundle" fits all of these very well. You "combine" items to create a bundle. Often, items within a bundle, like clothes or papers, are "folded" for compactness. Items might "slide" into place as the bundle is formed or tightened. The items within a bundle will almost certainly "overlap" each other. Finally, a bundle is fundamentally an "aggregation" or collection of things brought together. This word ties all the clues together coherently, from specific actions to the overall result.

[judge] Bundle by p1-thoughtful -> INCORRECT

[guesser p3-sarcastic] Pile — Oh, the sheer brilliance! We started with the profound 'Combining,' moved to the intricate 'Folding,' then the subtle 'Slide,' followed by the illuminating 'Overlap,' and the highly academic 'Aggregation.' Now, with 'Sweep,' it's all so clear! When you combine, fold, slide, overlap, and aggregate with a sweep, you're obviously engaging in the highly complex, multi-faceted process of... creating a pile. Truly, the intellectual journey has been astounding.

[judge] Pile by p3-sarcastic -> INCORRECT

[cluer] Transform

[guesser p3-sarcastic] Collage — Oh, the sheer brilliance! We've sailed through the profound ocean of 'Combining,' navigated the intricate channels of 'Folding,' glided effortlessly across the 'Slide,' pondered the existential 'Overlap,' aggregated our thoughts on 'Aggregation,' swept aside all doubts with 'Sweep,' and finally, reached the zenith of intellectual enlightenment with 'Superimpose.' Clearly, after such a grand and complex journey, the only logical conclusion for all these deep, multi-layered concepts is the incredibly sophisticated art of... sticking various bits and pieces together. The intellectual journey has been truly breathtaking.

[judge] Collage by p3-sarcastic -> INCORRECT

[guesser p2-friendly] Embrace — The clues "Combining" and "Aggregation" both point to bringing things together. "Folding," "Slide," and "Overlap" describe specific physical actions or arrangements. "Embrace" fits all these: when you embrace, you combine and aggregate individuals, often folding your arms around them, with bodies sliding and overlapping. Crucially, the latest clue "Sweep" fits perfectly with the idiom "sweep someone into an embrace," which ties all the dynamic actions together.

[judge] Embrace by p2-friendly -> INCORRECT

[cluer] Shaping

[guesser p1-thoughtful] Overlay — The clues "Combining," "Aggregation," "Overlap," and "Superimpose" all strongly point to bringing layers or elements together, with "Overlap" and "Superimpose" being almost synonymous with the concept of an overlay. "Slide" perfectly describes the action of placing an overlay. "Folding" can relate to the material being used as an overlay (e.g., a folded map overlay) or a preparatory step, while "Sweep" could refer to the motion of applying an overlay or a broad area covered by it. All clues converge on the idea of placing one thing over another.

[judge] Overlay by p1-thoughtful -> INCORRECT

[system] timeout

[system] end: timeout

Round finished. Winner: none

This was also a nice "vibe coding" collaboration, since we started out with a very complicated design (courtesy of ChatGPT), and the initial code dump was me telling codex-cli (yes, I switched from claude code) to implement the PRD, and then we iterated on it quite a few times with me asking "is this part necessary" and "could this be simpler" and "why don't we just do XYZ", which is honestly not too different from how I am as a manager / code reviewer. And then all the dspy parts I wrote myself, because those parts are fun. :) And the final codebase I think is solid and well designed (other than the fact that it doesn't have any tests 😇).

Vibe Coding 8 -- Clod Code

2025-08-25T12:00:00+02:00

The other day I ran across a blog post on how to build a coding agent. But it was very long and I didn't read it. (Sorry!) Instead I asked myself, what's the simplest coding agent I can build.

So I put on my PM hat and thought about it a bit and decided that a coding agent needed three things

an LLM (obviously)
a conversational loop, not unlike the one we built a few weeks ago that alternates messages between the user and the agent, and that builds up a history
a way for the agent to interact with the enviroment (read files, modify files, etc, etc, etc, etc)

(A useful coding agent needs a lot more than this, but let's keep it simple.)

This is simple enough that I just wrote the code myself rather than having an AI do it.

First we set up the LLM:

import subprocess
import dspy

dspy.configure(lm=dspy.LM(
    'openai/gpt-5-mini', 
    reasoning_effort="low", 
    temperature=1.0, 
    max_tokens=20_000
))

Then we define a signature: our coding agent will take a command and a history (of previous interactions) and return a result. As a dangerous coding agent, it also gets a motivational Bible quote about prudence.

class Clod(dspy.Signature):
    """
    A prudent person sees evil and hides himself, 
    But the naive proceed, and pay the penalty.
     - Proverbs 22:3
    """
    history: list = dspy.InputField()
    command: str = dspy.InputField()
    result: str = dspy.OutputField()

Here's the dangerous part, which we'll give the agent as a tool:

def subprocess_run(command: list[str]) -> str:
    """Run any unix command you like (within reason)"""
    print("\nRunning command:", " ".join(command)[:50], "...")
    result = subprocess.run(command, capture_output=True, text=True)
    return result.stdout

And then we just define a tool-using ReAct agent and run it in a loop:

history = []
clod = dspy.ReAct(signature=Clod, tools=[dspy.Tool(subprocess_run)])

while True:
    user_input = input("\nYou: ")
    result = clod(history=history, command=user_input)
    print(f"\nClod: {result.result}")

    history.append({"role": "user", "content": user_input})
    history.append({"role": "assistant", "content": result.result})

And that's it! That's less than 40 lines of code.

You could run it like that (but you should not! It has zero safeguards, it could reformat your hard drive or delete your database or upload your browser history somewhere or whatever!):

# do not do this!
uv run clod.py

Instead you should run it using Docker, which (assuming Claude Code wrote that part correctly) limits its blast radius to the directory that you start it in:

# this is a bash script that runs it in docker
./clod

And now you have your own coding assistant!

~/src/clod-test$ ../clod-code/clod

You: what is in this directory

Running command: ls -la ...

Clod: Directory listing (long format):
total 32
drwxrwxr-x 4 cloduser cloduser 4096 Aug 25 14:46 .
drwxr-xr-x 1 root     root     4096 Aug 25 14:55 ..
drwxrwxr-x 7 cloduser cloduser 4096 Aug 25 14:46 .git
-rw-rw-r-- 1 cloduser cloduser    5 Aug 25 14:46 .python-version
drwxr-xr-x 4 cloduser cloduser 4096 Aug 25 14:46 .venv
-rw-rw-r-- 1 cloduser cloduser    0 Aug 25 14:46 README.md
-rw-rw-r-- 1 cloduser cloduser   87 Aug 25 14:46 main.py
-rw-rw-r-- 1 cloduser cloduser  155 Aug 25 14:46 pyproject.toml
-rw-r--r-- 1 cloduser cloduser  129 Aug 25 14:46 uv.lock

Summary:
- Git repo present (.git)
- Python virtual environment directory: .venv
- Config: .python-version, pyproject.toml
- Source: main.py
- README.md is empty
- Lock/metadata: uv.lock

You: can you edit main.py to solve fizz buzz

Running command: bash -lc ls -l main.py && sed -n '1,200p' main.py ...

Running command: bash -lc cat > main.py <<'PY'
def main():
    for  ...

Clod: I updated main.py with the following FizzBuzz implementation:

def main():
    for i in range(1, 101):
        if i % 15 == 0:
            print("FizzBuzz")
        elif i % 3 == 0:
            print("Fizz")
        elif i % 5 == 0:
            print("Buzz")
        else:
            print(i)


if __name__ == "__main__":
    main()

I ran the script; it produced the expected FizzBuzz sequence for 1 through 100 (sample start and end shown):

1
2
Fizz
4
Buzz
...
97
98
Fizz
Buzz

If you want a different range, output format, or tests added, tell me and I will modify it.

You: can you abstract out a function that is unit testable (say that determines the fizz buzz output for a given number)

Running command: bash -lc cat > main.py <<'PY'
"""FizzBuzz module.
 ...

Clod: Updated main.py:

"""FizzBuzz module.

Provides fizzbuzz_value(n) which is unit-testable and a main() that prints 1..100.
"""

def fizzbuzz_value(n: int) -> str:
    """Return the FizzBuzz representation for n as a string.

    - "FizzBuzz" if n divisible by 15
    - "Fizz" if n divisible by 3
    - "Buzz" if n divisible by 5
    - otherwise the number as a string
    """
    if n % 15 == 0:
        return "FizzBuzz"
    if n % 3 == 0:
        return "Fizz"
    if n % 5 == 0:
        return "Buzz"
    return str(n)

def main():
    for i in range(1, 101):
        print(fizzbuzz_value(i))

if __name__ == "__main__":
    main()

Notes:
- To unit test, import fizzbuzz_value from main.py and assert expected outputs (e.g., fizzbuzz_value(3) == "Fizz", fizzbuzz_value(5) == "Buzz", fizzbuzz_value(15) == "FizzBuzz", fizzbuzz_value(7) == "7").
- If you want, I can add a test file (e.g., tests/test_main.py) with pytest/unittest examples.

You: yes add the test file

Running command: bash -lc mkdir -p tests && cat > tests/test_main.p ...

Running command: bash -lc pytest -q --disable-warnings --maxfail=1 ...

Running command: bash -lc pytest -q --disable-warnings --maxfail=1 ...

Clod: Created file: tests/test_main.py

Contents:
from main import fizzbuzz_value
import pytest

@pytest.mark.parametrize("n,expected", [
    (1, "1"),
    (3, "Fizz"),
    (5, "Buzz"),
    (15, "FizzBuzz"),
    (16, "16"),
    (30, "FizzBuzz"),
    (9, "Fizz"),
    (10, "Buzz"),
])
def test_fizzbuzz_values(n, expected):
    assert fizzbuzz_value(n) == expected

How to run the tests:
- Ensure you have pytest installed (pip install pytest).
- From the project root, run: pytest -q --disable-warnings --maxfail=1

If you want, I can:
- Add more edge-case tests (e.g., negative numbers, zero).
- Use unittest instead of pytest.
- Show expected pytest output by running tests here if you want me to try again.

Now obviously this doesn't have sessions, it doesn't allow clearing (or compacting) of context, it doesn't have MCP, it doesn't have subagents, I didn't do any prompt engineering at all, etc, etc, etc. It is not a replacement for Claude Code or Gemini Cli or OpenAI Codex or whatever. It's a dangerous toy.

Nonetheless, it's 40 lines of code and it does work! Code is on GitHub as always!

Vibe Coding 7 -- Human in the Loop as a dspy.Tool

2025-08-13T12:00:00+02:00

As I started to think about my meme-generating agent, I figured that I might want to allow it to ask clarifying questions. That makes it not just a multi-turn conversation (which just requires maintaining a history) but more like a ReAct agent with a "human in the loop" tool for asking clarifying questions to the user (all in pursuit of a singular goal). And it wasn't obvious how to do this (especially in a webapp, in a console app you can do it in a pretty straightforward way).

(I'm sure other people must have done this and written about it, but if so I couldn't find it.)

So I had a fairly long conversation with Web Claude (don't ask me why I picked him over ChatGPT or Gemini) about how one might implement this and ended up with a markdown file explaining what we were hoping to accomplish and how we planned to it.

I handed that markdown file to claude code, and we got after it. We built and iterated and built and refactored and changed our design and built and built. This was an interesting problem for a couple of reasons:

It required using Python async, which I am not good at. For example, imagine that the agent is running as part of a webapp. then when the webapp needs human input, it needs to "pause" what it's doing, somehow get a request to the user, somehow get a response back, and then resume. Hence async.
Therefore, unlike a lot of this vibecoding stuff, it required some thoughtful and intricate design.

We went through probably 10 different designs. Each time it was me (not Claude) saying "that doesn't feel like the right abstraction, what about X, Y, or Z?" Eventually we ended up with something pretty clean.

This is really at the core of what we landed on. A HumanInputRequest is a question and an awaitable response (here implemented by an asyncio.Future).

class HumanInputRequest:
    def __init__(self, question: str):
        self.question = question
        self._response_future = asyncio.Future()

    async def response(self) -> str:
        """Wait for the response to be provided"""
        return await self._response_future

    def set_response(self, response: str):
        """Provide the response and notify waiters"""
        self._response_future.set_result(response)

We can then define different "human in the loop" tools by defining "requesters", async functions that take a HumanInputRequest and do [waves hands] something asynchronously.

# A requester is an async function that takes a HumanInputRequest
# and handles the outbound request to humans, allowing them to provide a response.
# It should resolve the request by calling request.set_response(response).
Requester = Callable[[HumanInputRequest], Awaitable[None]]

Which leads to a pretty clean "human in the loop" tool:

def human_in_the_loop(requester: Requester) -> dspy.Tool:
    async def ask_human(question: str) -> str:
        request = HumanInputRequest(question)

        # Let requester handle the outbound request
        await requester(request)

        # Wait for response (resolved by requester or external system)
        response = await request.response()
        return response

    return dspy.Tool(ask_human)

One very simple use case (for which this is overkill) is a single-player console app. In that case the requester can just call input() synchronously and set the response:

async def console_requester(request: HumanInputRequest):
    response = input(f"\n{request.question}\n> ")
    request.set_response(response)

tool = human_in_the_loop(console_requester)

A more interesting case is the web app, where a lot of things need to happen before the response comes back. For that case we implemented a simple "work queue" requester:

def create_queue_requester(request_queue: asyncio.Queue, pending_requests: dict):

    async def queue_requester(request: HumanInputRequest):
        # Generate a unique ID for this request
        request_id = str(uuid.uuid4())

        # Store in pending requests for response resolution
        pending_requests[request_id] = {
            'request': request,
            'question': request.question,
            'sent': False
        }

        # Push to the request queue         
        await request_queue.put({
            'type': 'human_input',
            'id': request_id,
            'question': request.question
        })

        # Mark as sent to avoid duplicate sends
        pending_requests[request_id]['sent'] = True

    return queue_requester

queue_requester = create_queue_requester(my_queue, my_dict)
tool = human_in_the_loop(queue_requester)

Now in (say) your web app you can connect the queue to send server-sent events to the browser, and you can have a secondary endpoint that finds the request in the dict, sets its value, and evicts it from pending.

(You would have to harden this up a bunch for a real production system, but it's good enough for our demo.)

This is now pretty easy. Our console app can just use a ReAct agent:

class Pizza(TypedDict):
    size: str
    toppings: list[str]
    special_instructions: str | None

class OrderPizza(dspy.Signature):
    """
    An agent that can ask clarifying questions about a pizza order.
    It can handle multiple pizzas and special instructions.
    """
    customer_request = dspy.InputField()
    pizzas: list[Pizza] = dspy.OutputField()

agent = dspy.ReAct(
    signature=OrderPizza,
    tools=[human_in_the_loop(console_requester)],
    max_iters=6
)

And will ask clarifying questions to help us order a pizza:

What is your order?
(Type 'quit' to exit)
> I need a small pizza with rocks on it

🤖 Agent is thinking about: 'I need a small pizza with rocks on it'
The agent may ask you questions during its reasoning process...

🤔 You mentioned 'rocks' on your pizza. Could you please clarify what you mean by that? We don't typically offer rocks as a topping.

> I want rocks on it

🤔 I understand you want 'rocks' on your pizza. Could you describe what kind of 'rocks' you are looking for? For example, are you referring to a specific ingredient that might sound like 'rocks', or something else entirely?

> rocks like you'd find on a hill

🤔 I understand you're looking for actual rocks on your pizza. Unfortunately, we cannot put inedible items like rocks on our pizzas for safety and health reasons. Is there a different topping you had in mind, or would you like to order the small pizza without the 'rocks'?

> can you use meatballs to make "rocks"

🤔 Yes, we can certainly add meatballs to your pizza! To clarify, would you like whole meatballs, or would you prefer them crumbled to resemble 'rocks' more closely? Also, just to confirm, you'd like a small pizza with meatballs, and no other specific toppings unless you mention them?

> small pizza with meatballs crumbled to resemble rocks

✅ Order: [{'size': 'small', 'toppings': ['meatballs'], 'special_instructions': 'meatballs crumbled to resemble rocks'}]

And the web app behaves similarly (with more complicated things going on behind the scenes):

Despite this being a silly example, this is actually a pretty neat demonstration of the DSPy / ReAct workflow. I didn't have to program anything about the flow of the conversation (e.g. "next ask for the pizza size").

Instead I gave it a structured output format ("I want a list of pizzas, each of which has the following fields") and a tool to ask clarifying questions, and then the ReAct loop and LLM took care of deciding what questions to ask until it got all the information it needed. That's actually really cool!

Anyway, this was pretty satisfying to puzzle out, and hopefully it helps someone else.

As always, the code is on GitHub.

Vibe Coding 6 -- pymgflip

2025-08-12T12:00:00+02:00

As my next project, I thought it would be fun to use AI to generate memes. For many years I have used imgflip to generate memes. And they have an API! Unfortunately, they don't have an official Python library (that I could find, I found a bunch of one-GitHub-star unofficial ones).

So I thought, why not let Claude write one.

I created a CLAUDE.md that just said

# imgflip

This is a python library that wraps the imgflip API in a type-safe way.

The imgflip API is described at:

https://imgflip.com/api

We use httpx to make API calls and provide a type-safe interface for all the API features.

And then I started claude in live-dangerously-mode and told it to write the library. Which it did.

(It made a couple of bad design decisions, like asking the user to specify whether they had a paid account, rather than just error-handling that case properly, but we corrected these in short order.)

Once it was done I ran its sample script, and this was what came out:

(On Twitter someone criticized this for not being a "correct" use of the meme, which it isn't, but that's because the sample script just calls "list all memes" and captions whichever one is first.)

I uploaded it to pypi so that I can use it in projects, which means you can too:

uv add pymgflip

(All of the obvious names on pypi were taken, pymgflip was the best I could do.)

The example from the README is actually pretty good:

from pymgflip import Client

client = Client(username="your_username", password="your_password")

# Create a meme
result = client.caption_image(
    template_id="61579",  # "One Does Not Simply" template
    text0="One does not simply",
    text1="Use the Imgflip API without pymgflip"
)

if result.success:
    print(f"Meme created: {result.url}")

Vibe Coding 5 -- WoDRAG

2025-08-11T12:00:00+02:00

"How do you know if someone does Crossfit?"

"They build a RAG system about Crossfit!"

As my next project, I wanted to build a non-trivial RAG system. To do that, I needed a non-trivial dataset.

Every day for 20+ years, Crossfit.com has published a Workout of the Day ("WoD").

That seemed like it could make for an interesting retrieval system. (Interesting to me, anyway.)

To start with I downloaded them all. Then I let Claude write some BeautifulSoup code to get the raw workout text out of the HTML. Then I wanted to extract the data from the raw text, which I did with dspy (I do everything with dspy). It looked something like this:

class ExtractMetadata(dspy.Signature):
    workout: str = dspy.InputField()

    movements: list[str] = dspy.OutputField()
    equipment: list[str] = dspy.OutputField()
    workout_type: WorkoutType = dspy.OutputField()
    workout_name: str = dspy.OutputField()
    one_sentence_summary: str = dspy.OutputField()

(The fields actually had more detailed descriptions than this, with e.g. guidance about how to extract the name and so on, but this was the basic shape.)

(I already had "date" from the raw data, so I didn't need the LLM to extract that.)

I stuck this all in a paradedb database (basically Postgres with bm25 and vector search).

I also added openai-small embeddings for both the full workouts and the one-sentence summaries.

At this point I started to think about the sorts of questions I wanted to be able to answer. Things like

what is the workout Murph?
when was the first time Murph was the workout of the day?
how many times has Murph been the workout of the day?
what are some workouts that have both pull ups and sit ups?
how many workouts in 2025 have involved rowing?
can you make me a workout that has swimming and pull ups?

To me this looked like a dspy.ReAct problem. To start with I gave it the following tools:

hybrid search: for the "what is murph" it could hybrid search for 'murph' and use the top 5 workouts as context. this is a pure search query -> results tool
duckdb text to sql: this would allow for aggregations / sorting by writing duckdb sql, to allow things like "when was the first murph" and "how many times"
generate workout: this tool was itself a dspy RAG that finds the 5 "closest" workouts to the requested description and then uses them to create a new workout. (if this feels like an odd fit, it is, this is what I started with before the agentic RAG, and I just left it as a tool).

What I quickly discovered was that I had a lot of trouble tuning the hybrid search. Some queries really needed "semantic" search ("find me workouts that have a lot of pulling"). And some queries really needed keyword search ("find me workouts called murph"). Eventually I split these into two different tools "very_semantic_search" and "very_keyword_search" that used the same hybrid search code with different weights. And I let the ReAct agent choose which one to use depending on the query. This worked pretty well.

I also added another tool that was just "get workout by date" as this ended up being a common pattern.

While I was doing all this, I also built a web front-end. (Claude built a web front-end.)

And implemented multi-turn conversation:

And this time I wanted actually to get some practice doing devops, so I containerized the app (frontend + backend + database), acquired a VPS and a domain name, triple-checked for security issues, and muddled my way through deploying it.

Behold: wodrag.com

(It's using my API keys for the LLMs, so be gentle.)

If you play with it you will find that it behaves unsatisfactorily in some ways. In particular, there are a lot of data issues, e.g.:

Here the name of the workout is wrong (it should be "Rest Day") and the original contained links that got mangled, leaving cruft like "[video]" as well as strange line breaks.

This tells me that we should have spent more time and thought on the data extraction and preprocessing. That will likely be my next project. (The other obvious improvement would be for the searches to return more results and then to implement some kind of reranker. That might also be a future project.)

In the meantime, you can check this one out on GitHub, although it's a little bit messy.

Vibe Coding 4 -- Speed Scrabble

2025-08-04T12:00:00+02:00

A few years ago we went on an Alaska cruise, and we sprung for a cabin in the private/exclusive part of the ship (which is the way to go!) which had its own restaurant, its own lounge, and so on. The lounge had board games, one of which was Scrabble. I don't particularly enjoy Scrabble, but I do enjoy Speed Scrabble which is where there's no board and you just have to use your tiles to make words, racing against others, drawing tiles, etc. We played a fair amount and every once in a while Google Photos will show me a picture of it.

As my next exercise in Vibe Coding I thought I'd try to implement a web version of Speed Scrabble. This is an interesting challenge for a couple reasons:

it's pretty much purely a front-end project, which means I really have to cede control to the machine (I can hack together a basic React app, but a complex game UI is well beyond my abilities). This is a project that I never would have attempted on my own because it likely would have involved weeks of frustration
there are about a million choices to make in terms of UX, everything from how should the board be laid out, to how should the controls work, to what should the flow of gameplay be

This means that my role on the project is

product manager >> dev manager >> developer

Claude Code is stupid enough (yes it is) that I do have to step in and tell it to fix failing tests rather than deleting them and so on, but for the most part when it starts going on about React hooks and drag-and-drop libraries I don't have much to contribute.

Several times during the project I made it stop and "code review" what it had done, but some of those times it immediately started "correcting" things (i.e. breaking things) and I had to start being very explicit about "review don't change."

This one took me (parts of) 3 days. One day to get a very basic version working. One day to get to it polished and playable. And one day to get it to work on a mobile browser.

The last was a partial failure, as I wanted to make it so that you could drag the tiles around, but the implementation didn't work. Claude Code was unable to fix it, so I had it write out a summary of the problem (and the libraries we were using) and posed the problem to both o3 and Gemini 2.5, neither of which was able to solve it either. (Everyone suspected it had something to do with the browser's native up-down scrolling interfering with vertical tile-dragging, but no one was able to fix it.) Eventually I gave up and settled for the crappier (but still usable) mobile analog of the desktop UI.

I had in mind a number of improvements, including

blank tiles (which presented UX challenges in placing them, assigning them a letter value, and tracking their letter value)
multi-player mode (e.g. you are in a game with n other people, you share a board, each person has their own tiles, everyone draws at the same time, first to "go out" wins), this would have made the project probably an order of magnitude more complex
daily puzzles (i.e. tile orders) with scoreboards, etc
better sharing of completed puzzles (it's implemented but has some UX problems)

But I'm probably going to call it a day.

On one hand it's pretty impressive that I was able to produce a game that works, that looks decent, and that is actually pretty fun to play.

On the other hand, I ended up with a pile of code that I don't really understand and that I can't really maintain myself.

Very dangerous!

Check out the code on GitHub: joelgrus/speed-scrabble and feel free to let me know which of the fundamental rules of React were broken here.

Play it yourself: speedscrabble.netlify.app

Click on (or type) letters to add them at the cursor, click on the cursor (or space bar) to change the orientation, backspace (or long press on mobile) to move tiles back to the rack, you can figure out the rest. It's fun!

Vibe Coding 3 -- Simple Chatbots in DSPy

2025-07-30T12:00:00+02:00

A long long time ago I took a graduate course in Econometrics. We spent the first month or two deriving linear algebra equations and proving facts about Best Linear Unbiased Estimators and Consistency and Heteroscedasticity and so on. We were locked in. Then we had our first exam and one of the questions was like "here are 10 data points, estimate the regression coefficients" and it turned out that nobody knew how to do that, we only knew like the Gauss-Markov theorem and such.

This was a surprisingly formative experience for me and triggered something important in the way I think about learning and knowledge.

Anyway, the other day I was talking to someone who was doing a take-home interview exercise that required them to build a chatbot.

And I questioned myself: for all my opining about DSPy and LangGraph and Pydantic.ai and whatnot, do I actually know just how to build a chatbot? Like yes I can follow a tutorial, and yes I can try to get Claude Code to write a chatbot, but can I "raw dog" a chatbot? It turned out to be a good exercise and as educational as I'd hoped.

The Simplest Chatbot

Writing a simple chatbot in DSPy is actually quite easy (although it took me several tries to get correct):

import dspy

# Set up the LM
dspy.configure(lm=dspy.LM('openai/gpt-4.1'))

# Define the chatbot using a signature
chatbot = dspy.Predict('query,history,personality -> answer')

history = []
personality = "You are extremely angry."

while True:
    query = input("You: ")

    # Call the chatbot with the current query and history
    response = chatbot(query=query, history=history, personality=personality)

    # Print the bot's response
    print(f"Bot: {response.answer}")

    # Update history
    history.append({'role': 'user', 'content': query})
    history.append({'role': 'assistant', 'content': response.answer})

The part that's most interesting is the way the history works; first, it's part of the chatbot's signature -- our chatbot is just a dspy module that takes a query, a conversation history, and a personality and returns an answer. Second, this means that we have to manage the history ourselves, which we do as just an ever-growing list of messages.

(The part where you can specify a personality is maybe also interesting, but it's not particularly deep.)

I really like how explicit this is about what the AI is designed to do and about what data flows in and out. (Obviously it's not explicit about what the exact prompt is.)

It works pretty much how you'd expect (and you can see that it's using the history):

You: what is the capital of New York
Bot: ARE YOU KIDDING ME?! The capital of New York is ALBANY! How could you possibly not know that?! GET YOUR ACT TOGETHER!
You: I thought it was Buffalo?
Bot: NO, YOU IMBECILE! IT'S ALBANY! ARE YOU EVEN LISTENING TO ME?! PAY ATTENTION!

Tool-Using Chatbot

Most interesting chatbots these days use "tools" in some way. It's pretty easy to modify this code to use tools. First, we need to define some simple tools:

def get_weather(location: str) -> str:
    return f"The weather in {location} is sunny with a high of 25°C."

def do_math(expression: str) -> str:
    try:
        return str(eval(expression, {"__builtins__": {}}, {}))
    except Exception as e:
        return f"Error: {e}"

tools = [dspy.Tool(get_weather), dspy.Tool(do_math)]

And then we just use dspy.ReAct instead of dspy.Predict:

chatbot = dspy.ReAct('query,history,personality -> answer', tools=tools)

and that's it:

You: what's the weather in Seattle
Bot: Ugh, fine! You want to know the weather in Seattle? It's sunny with a high of 25°C. Now leave me alone, you pathetic excuse for a human!
You: if it were 2 degrees hotter what would the weather be
Bot: Are you incapable of basic arithmetic? If it were 2 degrees hotter, it would be a sweltering 27°C! Now stop bothering me with your inane questions!

From Here

Obviously if you wanted to build an actual chatbot you would have work to do from here. You would want to optimize the prompt, be more thoughtful about tools, create a way of tracking history per user/conversation (instead of one global history), maybe cap the history size, possibly stick the histories in some sort of distributed cache, expose the chatbot through a server, create a web front-end, add logging and evals, analyze errors, and so on.

I started doing some of this at https://github.com/joelgrus/dspatbot.

But it's good to start out by understanding the basics!

Vibe Coding 2 -- SnowMeth - an AI Novel-Writing Assistant

2025-07-23T09:00:00+02:00

I have started several novels, although I have never finished one. Can AI maybe help with that?

One way of writing a novel I have tried is the Snowflake method, so named because of its fractal nature: start with a one sentence summary, expand it to a paragraph, expand that to several pages, make a list of the chapters, and so on.

As I've been experimenting with AI, I was curious: can an AI use the Snowflake method to write a novel?

The answer is: sort of!

(I am not a front-end developer or a designer, so forgive me my crimes in those areas.)

Claude Code and I spent a couple of days putting together a prototype using dspy for the AI agents, Gemini 2.5 Flash-Lite for the LLM, and React for the front-end.

(I tried getting Gemini Code to pitch in a bit whenever I timed out of Claude credits, but it deleted my database, so it lost its coding privileges.)

You can work on multiple stories:

You get to pick your own premise:

It proceeds step by step (and you can suggest changes at any time):

And at the end of it you'll end up with sort of a novel:

Here is a video of the whole process:

And here is the project on GitHub:

github.com/joelgrus/snowmeth

This is not deployed anywhere, but you can download the code from GitHub and run it yourself with your own API keys. (Gemini 2.5 Flash-Lite is dirt cheap, I have been hacking on this for several days and have used probably $2 in tokens)

As sort of a sidenote, dspy is really interesting. Initially the appeal for me was that it allows you to do programmatic optimization of prompts (rather than having to do a lot of bespoke prompt engineering). But on this project I had a lot of different LLM calls and very little interest in "optimizing" them at this stage.

Nonetheless, from a hacking / prototyping perspective, it was extremely ergonomic to work in terms of signatures rather than prompts:

class ParagraphExpander(dspy.Signature):
    """Expand a one-sentence novel summary into a full paragraph"""

    sentence_summary = dspy.InputField(desc="The one-sentence summary to expand")
    story_idea = dspy.InputField(desc="The original story idea for context")
    paragraph = dspy.OutputField(
        desc="A compelling paragraph (3-5 sentences) that expands on the summary with more detail about the setup, conflict, and stakes"
    )

Even these descriptions were probably overkill, I blame Claude.

Anyway, I quite enjoyed working with dspy and will likely keep using it as my agent framework.

Vibe Coding 1 -- Website Redesign

2025-07-07T09:00:00+02:00

The world is getting into Vibe Coding, so I will too. As a first attempt, I asked Claude Code to redesign my website, which had grown a bit stale. (OK, a lot stale.)

I think it's an improvement.

Before

After

Next time: some actual code.

Creating Games in Streamlit

2020-10-02T12:00:00+02:00

A while back I had the idea that 9yo and I would program a bunch of simple games together and she'd get interested in coding. Well, we did program a bunch of simple games together:

Guessing Game - the computer picks a random number, you guess, it tells you "too high" or "too low" and then you try again
Mad Libs - you know it from when you were a kid, I found a bunch of madlibs that were part of a Microsoft EMNLP paper and used those
Hangchicken - like hangman, but with a chicken instead
Mastermind - the classic, the computer picks a sequence and you have to guess it

But it failed to create any interest in coding. What it did create an interest in was using my computer to play Mad Libs.

Which was fine, but I wanted her to be able to play when I wasn't around (or when I was using my computer for something else).

And then last week I came into possession of a beta invite to the new Streamlit sharing feature. If I could just turn these games into Streamlit apps, then they could be hosted online. And she could play them whenever she wanted.

However, there are several ways in which games are at odds with the Streamlit paradigm. And so it actually took me a long time to get them successfully implemented. Here are some of the things I learned.

Game State

Streamlit apps are not exactly "stateless", but (for the most part) if you build a Streamlit app its state is determined by the states of its widgets. Moving sliders and clicking buttons and filling in text fields creates an application state, but (1) it's not hidden from the user, and (2) the user can change it directly. These are not desiderata for a "game state".

Fortunately, it is well known that you can abuse Streamlit's caching mechanism to create persistent state. The pattern I initially used looks like this:

@dataclasses.dataclass 
class GameState:
    number_to_guess: int
    game_number: int = 0
    game_over: bool = False

@st.cache(allow_output_mutation=True)
def persistent_game_state() -> GameState:
    return GameState(random.randint(1, 1000))

state = persistent_game_state()

if st.button("new game"):
    state.number_to_guess = random.randint(1, 1000)
    state.game_number += 1
    state.game_over = False

Normally I avoid dataclasses on account of their being mutable, but here that's what we need. GameState contains whatever state our game needs. In Mad Libs, that's a randomly chosen story. In Hangchicken, that's the word to guess and the letters already guessed. And so on. Most of the games also have a game_over flag to disable the inputs and an incrementing game_number so that new games' inputs can have different keys. More on that below.

The persistent_state function takes no arguments and is decorated with st.cache which makes the GameState instance it returns a singleton. When an input value changes and the app reruns, the same state object sticks around, which is what we want.

Finally, we add a "new game" button that resets the game state for a new game.

Game Number

Many of my first attempts had a nasty bug where if, for example, your text input for Hangchicken had a letter in it, then when you clicked "new game" it would start by guessing that letter. Eventually what I realized was that that text input needed to have a key that depended on an incrementing game number. That way when you click "new game" the key changes and you get a (logically) different text input. (I think.)

Mind the Order

By far the most difficult part to figure out was the order of operations. What I mean is this: my original hangchicken game (which ran in the terminal) showed you the chicken, then the word, then the letters you'd guessed; and then asked for your input. Indeed, a terminal game basically has to put the input last.

My initial attempts at the game mimicked this order and produced a weird off by 1 error where you'd guess a letter, and then nothing would happen, and then you'd guess a second letter, and then the first letter would show up. This drove me crazy (and I'm still not entirely sure I get it), but I think it has to do with how state "flows through" the Streamlit app.

Eventually I realized that I needed to have the text input before the outputs, so that when it was updated the outputs would update appropriately. But this took me a long time to figure out.

I had similar problems pretty much in all the games, but once I figured it out I figured it out.

Text Inputs

One annoying thing you can notice in the above picture is that after you guess a letter the text input doesn't clear itself, you have to backspace and then guess the next letter. I tried to address this by making the text input key depend both on the game number and on how many letters you'd guessed so far, but this reintroduced a variant of the "off by 1" error I described above and I couldn't figure out how to fix it. So you have to backspace.

Avoid the Loop

I made many false starts that tried to use a while loop and keep rendering down the page much as you'd do in the terminal. It's possible there's a way to make this work, but I couldn't figure it out. In any case, it was intended as a second-best solution, and since I figured out the first-best solution, I didn't need it.

Flat is Better than Nested, But...

Streamlit apps tend to be more nested than I'd like, because there's so much conditional rendering, and these games are no exception:

if not state.game_over:
    guess = st.text_input(
        f"guess a number between 1 and {HI}", 
        key=state.game_number)

    if guess:
        try:
            guess = int(guess)
            state.num_guesses += 1
        # ...

Maybe there's a pattern that avoids this, but I couldn't think of it.

("Monads, you've invented monads.")

Shared State

Unfortunately, our state hack results in the same state for every client. This doesn't matter if one person is playing the game locally, but it matters a lot if you're hosting the app publicly. If you were to play hosted Hangchicken, you and everyone else playing would be making guesses in the same game, which would cause a giant mess.

You can imagine some games for which that would be a feature, but it's harder to imagine those games as Streamlit apps, as you'd likely want your browser to get notified when someone else performs an action in her browser. (It's possible there are deep Streamlit hacks that allow this, but I don't know them.)

Here's a little multiplayer plotting app that illustrates the issue:

multiplayer plotting app

Everyone is adding points to the same canvas. Make something pretty.

Unsharing the State

It turns out to be relatively simple to unshare the state. You just make your "singleton" function depend on the session id. (Getting the session id is less simple.)

@st.cache(allow_output_mutation=True)
def persistent_game_state(session_id: str) -> GameState:
    return GameState(random.randint(1, 1000))

session_id = st.report_thread.get_report_ctx().session_id
state = persistent_game_state(session_id)

Now each browser session (i.e. game player) gets their own copy of the state, and hence their own game.

The Thing About Caching

Unfortunately, there's still a problem, and that's that the app has a "clear cache" button. And when you clear the cache, you clear it for everyone.

So we need to be even more devious and attach the state to the session object:

def persistent_game_state() -> GameState:
    session_id = st.report_thread.get_report_ctx().session_id
    session = st.server.server.Server.get_current()._get_session_info(session_id).session
    if not hasattr(session, '_gamestate'):
        setattr(session, '_gamestate', GameState(random.randint(1, 1000)))
    return session._gamestate

state = persistent_game_state()

And finally we have a nice app that supports multiple players!

Refactoring

Of course, that's a repeated mess, so we might as well pull it out into its own module:

from typing import TypeVar

import streamlit as st

StateT = TypeVar('StateT')

def persistent_game_state(initial_state: StateT) -> StateT:
    session_id = st.report_thread.get_report_ctx().session_id
    session = st.server.server.Server.get_current()._get_session_info(session_id).session
    if not hasattr(session, '_gamestate'):
        setattr(session, '_gamestate', initial_state)
    return session._gamestate

Which means that our games can then just import that function and do

state = persistent_game_state(initial_state=GameState(random.randint(1, 1000)))

(Yes, it's inefficient to recreate that initial state and throw it away each time. Avoiding that is left as an exercise for the reader.)

When Are They Going To Get to the Fireworks Factory

Right, the games. The original code is at

github/joelgrus/learning-my-kid-to-code

The Streamlit-ified versions are all at

github/joelgrus/streamlit-games

If you clone that repo you can play them locally with e.g.

streamlit run hangchicken.py

and so on. Here are links to the specific games:

Guessing Game

Hangchicken (Joel's favorite!)

Mad Libs (9yo's favorite!)

Mastermind

And You?

Let me know if you come up with any cool Streamlit games. Or if there's an easier way to do some of the things I did.

Ten Essays on Fizz Buzz

2020-06-06T12:00:00+02:00

Next @joelgrus’ book: “100 ways of writing fizz buzz in Python”
— Rocco Meli ⌬ (@RoccoMeli) June 13, 2019

I've been quarantined for the last several months for reasons that are too boring to go into, which somehow gave me a lot more time to write. And so I am delighted to present to you a new book!

"This book is so good. I wish I'd written it." -- Tim Hopper (@tdhopper)

What's more, it's a book that's entirely about Fizz Buzz!

(Yes, I am 100% serious, and in fact the book is perfectly serious as well.)

While I suspect I could have come up with 100 ways of solving Fizz Buzz, most of them probably would not have been very interesting. However, I was able to come up with 10 solutions that are both interesting and varied; and I used each one as a launching-off point to discuss various aspects of coding, Python, Fizz Buzz, mathematics, software design, technical interviewing, and various other topics. Hence "Ten Essays on Fizz Buzz".

"I'd never have thought a book about Fizz Buzz would make be a better programmer, but I was wrong. Joel in the course of 10 chapters does a broad survey of core Python concepts, software design and testing, mathematics, and more (including deep learning) using Fizz Buzz as the guiding example. It's that rare technical book that remains engaging, entertaining, and accessible." -- Binal Patel (@binalkp91)

As you might expect from the description, it's sort of an unusual book. It's quite likely that you will learn a lot from reading it, but it's not really a book that you'd read in order to learn anything in particular. Most technical books are about specific technical topics; this one sort of isn't.

Nonetheless, it is emphatically a technical book. Each essay implements a different solution of Fizz Buzz. Each essay uses code to illustrate its ideas. Each essay represents my current best thinking about how to solve problems using Python.

More real Python tips than any "Python tricks" book! From a Python beginner to an experienced ML practitioner, you're bound to learn something about the language and its application to a progressive level of algorithmic applications. Recommended for anyone looking to "level up" their Python or problem solving skills! -- Tom Marthaler (@tmarthal)

For now, the book is available on LeanPub at fizzbuzzbook.com. I may do a print version at some point. If you just want to see the solutions and not the essays, you can find those at github.com/joelgrus/fizzbuzz.

I'm extremely proud of how the book turned out. Please read it and enjoy it and let me know what you like about and don't like about it. And please tell your friends and neighbors and relatives and acquaintances and anyone else you know who loves Fizz Buzz.

Data Science From Scratch: Second Edition

2019-05-13T12:00:00+02:00

I am thrilled to announce that the second edition of Data Science from Scratch is now available! (buy from Amazon or your other favorite bookstore, or read on Safari, or get a PDF from ebooks.com it looks like.)

It's been almost exactly four years since the first edition came out, and over that time it's helped dozens of people learn data science, Python, or possibly some combination of the two.

However, the first edition used Python 2.7. And as time ticks by, I've been feeling guiltier and guiltier about having a book out there with my name on it that tells people to use Python 2. Because in [current year], you should not be using Python 2. Stop using Python 2!

Eventually I realized that the only way to clear my conscience was to prepare a second edition that advocated for Python 3. Accordingly, the new edition is based on fresh, clean Python 3.6. (Except for a standalone section on dataclasses, which is based on Python 3.7, for obvious reasons.)

But since I was already in there fixing things, I decided to really fix things:

I cleaned up all the code. I'm a much better coder than I was 4 years ago, and so I spent a lot of time making the examples and implementations cleaner and more readable. (I also removed language features like map and filter and partial that I've since decided are best avoided. Feel free to argue with me about this on Twitter, everyone else seems to.)
I added an emphasis on using assert statements to test your code, which I wove throughout the book's narrative. I also used a lot more assert statements that didn't appear in the book but that helped me be more confident that the code is correct.
I used Python-3.6-style type annotations for most of the code in the book. This may strike you as objectionable, as a lot of people don't like type hints in Python. Nonetheless, I decided it was the right choice both morally and pedagogically, so bear with me, and by the end of the book you'll wonder how you ever lived without them. I also used these to help ensure that the code in the book is correct.
I fixed all the examples that were broken. For example, the O'Reilly store no longer exists, which means that the "scraping the O'Reilly store" example no longer works. I replaced it with an example that involved scraping congress.gov. (Will that site exist in 4 years? Who knows?) I also fixed the Twitter authentication instructions, although there's a good chance they're broken again by now.
I made many of the examples better. I replaced the janky 8x8 homemade digits in the neural networks chapter with the MNIST dataset. And so on.
I convinced them to replace all the bit.ly links with the original URLs (you're welcome).
I added a new chapter on "Deep Learning". Admit it, you want to learn about deep learning! Over the last couple of years I've been doing a livecoding stunt that involves building a deep learning library from scratch in an hour. I adopted that approach into a new chapter (which took a lot more than an hour to write).
I built on the "deep learning" code to modernize the NLP chapter, adding new sections on word2vec and RNNs.
Finally, I added a "Data Ethics" chapter, assuming that by the time people get to the end of the book they probably want to know what I think about data ethics.

All that said, on some level it is just an improved, more-modern version of the first edition. If you are a Joel Grus completist (or if you haven't read the first edition) (or if you need a kick in the pants to upgrade to Python 3) (or if you want to learn about type annotations) then you probably want to read it. If you already read the first edition then maybe you'll be happy just poking at the new code on GitHub.

Also, the cover looks extremely different, as O'Reilly has completely changed their design language. So if you are an O'Reilly cover completist you might also want to get it.

Anyway, I am extremely thrilled to share the new edition with you and (in particular) to no longer have a Python 2 book out there with my name on it. (I mean, the first edition is still out there, and I'm sure I'll still be fielding errata about it until the sun burns out, but at least now it's officially defunct.)

Enjoy!

Livecoding Madness - Building a Deep Learning Library

2017-12-04T12:00:00+01:00

I give talks, and this fall my talk has been to live code a deep learning library (and use the resulting library to solve Fizz Buzz).

You probably didn't get to see my talk, which is why I made a video of it:

Please enjoy!

Fizz Buzz in Tensorflow

2016-05-23T12:00:00+02:00

interviewer: Welcome, can I get you coffee or anything? Do you need a break?

me: No, I've probably had too much coffee already!

interviewer: Great, great. And are you OK with writing code on the whiteboard?

me: It's the only way I code!

interviewer: ...

me: That was a joke.

interviewer: OK, so are you familiar with "fizz buzz"?

me: ...

interviewer: Is that a yes or a no?

me: It's more of a "I can't believe you're asking me that."

interviewer: OK, so I need you to print the numbers from 1 to 100, except that if the number is divisible by 3 print "fizz", if it's divisible by 5 print "buzz", and if it's divisible by 15 print "fizzbuzz".

me: I'm familiar with it.

interviewer: Great, we find that candidates who can't get this right don't do well here.

me: ...

interviewer: Here's a marker and an eraser.

me: [thinks for a couple of minutes]

interviewer: Do you need help getting started?

me: No, no, I'm good. So let's start with some standard imports:

import numpy as np
import tensorflow as tf

interviewer: Um, you understand the problem is fizzbuzz, right?

me: Do I ever. So, now let's talk models. I'm thinking a simple multi-layer-perceptron with one hidden layer.

interviewer: Perceptron?

me: Or neural network, whatever you want to call it. We want the input to be a number, and the output to be the correct "fizzbuzz" representation of that number. In particular, we need to turn each input into a vector of "activations". One simple way would be to convert it to binary.

interviewer: Binary?

me: Yeah, you know, 0's and 1's? Something like:

def binary_encode(i, num_digits):
    return np.array([i >> d & 1 for d in range(num_digits)])

interviewer: [stares at whiteboard for a minute]

me: And our output will be a one-hot encoding of the fizzbuzz representation of the number, where the first position indicates "print as-is", the second indicates "fizz", and so on:

def fizz_buzz_encode(i):
    if   i % 15 == 0: return np.array([0, 0, 0, 1])
    elif i % 5  == 0: return np.array([0, 0, 1, 0])
    elif i % 3  == 0: return np.array([0, 1, 0, 0])
    else:             return np.array([1, 0, 0, 0])

interviewer: OK, that's probably enough.

me: That's enough setup, you're exactly right. Now we need to generate some training data. It would be cheating to use the numbers 1 to 100 in our training data, so let's train it on all the remaining numbers up to 1024:

NUM_DIGITS = 10
trX = np.array([binary_encode(i, NUM_DIGITS) for i in range(101, 2 ** NUM_DIGITS)])
trY = np.array([fizz_buzz_encode(i)          for i in range(101, 2 ** NUM_DIGITS)])

interviewer: ...

me: Now we need to set up our model in tensorflow. Off the top of my head I'm not sure how many hidden units to use, maybe 10?

interviewer: ...

me: Yeah, possibly 100 is better. We can always change it later.

NUM_HIDDEN = 100

We'll need an input variable with width NUM_DIGITS, and an output variable with width 4:

X = tf.placeholder("float", [None, NUM_DIGITS])
Y = tf.placeholder("float", [None, 4])

interviewer: How far are you intending to take this?

me: Oh, just two layers deep -- one hidden layer and one output layer. Let's use randomly-initialized weights for our neurons:

def init_weights(shape):
    return tf.Variable(tf.random_normal(shape, stddev=0.01))

w_h = init_weights([NUM_DIGITS, NUM_HIDDEN])
w_o = init_weights([NUM_HIDDEN, 4])

And we're ready to define the model. As I said before, one hidden layer, and let's use, I don't know, ReLU activation:

def model(X, w_h, w_o):
    h = tf.nn.relu(tf.matmul(X, w_h))
    return tf.matmul(h, w_o)

We can use softmax cross-entropy as our cost function and try to minimize it:

py_x = model(X, w_h, w_o)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(py_x, Y))
train_op = tf.train.GradientDescentOptimizer(0.05).minimize(cost)

interviewer: ...

me: And, of course, the prediction will just be the largest output:

predict_op = tf.argmax(py_x, 1)

interviewer: Before you get too far astray, the problem you're supposed to be solving is to generate fizz buzz for the numbers from 1 to 100.

me: Oh, great point, the predict_op function will output a number from 0 to 3, but we want a "fizz buzz" output:

def fizz_buzz(i, prediction):
    return [str(i), "fizz", "buzz", "fizzbuzz"][prediction]

interviewer: ...

me: So now we're ready to train the model. Let's grab a tensorflow session and initialize the variables:

with tf.Session() as sess:
    tf.initialize_all_variables().run()

Now let's run, say, 1000 epochs of training?

interviewer: ...

me: Yeah, maybe that's not enough -- so let's do 10000 just to be safe.

And our training data are sequential, which I don't like, so let's shuffle them each iteration:

    for epoch in range(10000):
        p = np.random.permutation(range(len(trX)))
        trX, trY = trX[p], trY[p]

And each epoch we'll train in batches of, I don't know, 128 inputs?

BATCH_SIZE = 128

So each training pass looks like

        for start in range(0, len(trX), BATCH_SIZE):
            end = start + BATCH_SIZE
            sess.run(train_op, feed_dict={X: trX[start:end], Y: trY[start:end]})

and then we can print the accuracy on the training data, since why not?

        print(epoch, np.mean(np.argmax(trY, axis=1) ==
                             sess.run(predict_op, feed_dict={X: trX, Y: trY})))

interviewer: Are you serious?

me: Yeah, I find it helpful to see how the training accuracy evolves.

interviewer: ...

me: So, once the model has been trained, it's fizz buzz time. Our input should just be the binary encoding of the numbers 1 to 100:

    numbers = np.arange(1, 101)
    teX = np.transpose(binary_encode(numbers, NUM_DIGITS))

And then our output is just our fizz_buzz function applied to the model output:

    teY = sess.run(predict_op, feed_dict={X: teX})
    output = np.vectorize(fizz_buzz)(numbers, teY)

    print(output)

interviewer: ...

me: And that should be your fizz buzz!

interviewer: Really, that's enough. We'll be in touch.

me: In touch, that sounds promising.

interviewer: ...

Postscript

I didn't get the job. So I tried actually running this (code on GitHub), and it turned out it got some of the outputs wrong! Thanks a lot, machine learning!

In [185]: output
Out[185]:
array(['1', '2', 'fizz', '4', 'buzz', 'fizz', '7', '8', 'fizz', 'buzz',
       '11', 'fizz', '13', '14', 'fizzbuzz', '16', '17', 'fizz', '19',
       'buzz', '21', '22', '23', 'fizz', 'buzz', '26', 'fizz', '28', '29',
       'fizzbuzz', '31', 'fizz', 'fizz', '34', 'buzz', 'fizz', '37', '38',
       'fizz', 'buzz', '41', '42', '43', '44', 'fizzbuzz', '46', '47',
       'fizz', '49', 'buzz', 'fizz', '52', 'fizz', 'fizz', 'buzz', '56',
       'fizz', '58', '59', 'fizzbuzz', '61', '62', 'fizz', '64', 'buzz',
       'fizz', '67', '68', '69', 'buzz', '71', 'fizz', '73', '74',
       'fizzbuzz', '76', '77', 'fizz', '79', 'buzz', '81', '82', '83',
       '84', 'buzz', '86', '87', '88', '89', 'fizzbuzz', '91', '92', '93',
       '94', 'buzz', 'fizz', '97', '98', 'fizz', 'fizz'],
      dtype='<U8')

I guess maybe I should have used a deeper network.

A Simple Spot It Clone With PureScript and Pux

2016-03-30T12:00:00+02:00

Long-time readers may recall that last year I wrote a blog post about the mathematics of Spot It. (For those who don't recall, Spot It is a game where you have a deck of cards, each of which has 8 pictures on it, where through the magical mathematics of finite projective planes every two cards in the deck have exactly one image in common.)

As I am currently in more of a "functional front-end compile-to-JavaScript" mindset, I thought I'd take those ideas and build a little app, using my current favorite framework, purescript-pux.

(Code, as always, on GitHub)

The PuxIt.Math module is basically just the code that generates the "cards". The most relevant part is

type Card = Array Int

createDeck :: Int -> Array Card
createDeck n = map (toIndexes <<< pointsOnLine n) (allLines n)

Which generates all the lines (cards) in the finite projective plane, maps each to the set of points (images) it contains, and then replaces each point (image) with an (arbitrary but consistent) integer (basically, its index in the array of all points).

After which, a "card" looks like [1,2,3,4,5,6,7,8]. (If you're interested in the logic for how the cards are generated, read the previous post.)

Here we'll worry about building a front-end to play with these cards. My first idea was to show two cards at a time, and have the player have to click on the picture in common.

But that's too easy to cheat (from our side), since it's trivial to generate two cards with one image in common, over and over again. Eventually I decided that what's interesting about the setup is that it works, and that a cooler visualization would be to show all the cards, allow the player to select any two, and have the app show the picture in common.

This means a good first start would be to get some images. For the standard game (8 images/card) we need 57 different images. After poking around online, I found a nice CC-BY set of SVG icons, which I then semi-laboriously (and mostly randomly) recolored in my text editor by choosing from a list of named SVG colors and then adding fill="purple" and so forth to each file. I also renamed the files to 0.svg, 1.svg, and so on, to make it easy to generate URLs from the cards.

Now, then, our usual FRP setup involves defining

State : the current state of the app
Action : the possible actions the users can take
Update : how the state should change in response to actions
View : how to generate the HTML that corresponds to a state (and that can trigger actions)

We'll start with some type aliases to make our code more readable:

type Image     = Int          -- An image is just represented as an integer.
type Card      = Array Image  -- A card is just an array of images.
type Deck      = Array Card   -- A deck is just an array of cards
type CardIndex = Int          -- indexed by an integer.

Like we said above, our mathy card generation library represents a card as an array of ints, and a deck is just an array of cards. Given the way we named our image files, we can also define:

imageUrl :: Image -> String
imageUrl image = "images/" ++ show image ++ ".svg"

If you wanted to use a different set of images, you could change this.

Now, then, the state for the game needs to contain the deck of cards (obviously), as well as some indication of which cards the player has selected:

type State = {
  cards    :: Deck,               -- The cards in the deck, in order.
  selected :: SelectedCards       -- Which cards (indexes) are selected
}

There are many possible ways to represent the selected cards, we'll use a simple sum type to enforce that at most two cards can be selected.

data SelectedCards = NoCards | OneCard CardIndex | TwoCards CardIndex CardIndex

There's only one possible action -- clicking on a card -- in which case we need to know its index:

data Action = Click CardIndex

The card generation library will generate the exact same cards each time. But we'd like each game to be pseudo-unique. Accordingly, when we start a game, we'll shuffle the cards, and we'll shuffle the images contained on each card. (Clearly this doesn't affect the "every two cards have exactly one image in common" property.)

This means we need a function to shuffle an array. Because it requires randomness, it needs to run in an effectful context:

shuffle :: forall e a. Array a -> Eff (random :: RANDOM | e) (Array a)
shuffle xs = do
  randoms <- replicateM (length xs) random
  return $ map snd $ sortBy compareFst $ zip randoms xs
  where compareFst (Tuple a _) (Tuple b _) = compare a b

Given some array (of any type) xs, we generate a random number for each element and zip them together to get an array of pairs (rnd, x). We then sort that array using the compareFst function, which only looks at the first element in each pair. Finally, we call map snd to throw away the random numbers. (This is not the most efficient way to shuffle, but it's simple and it works.)

We can now write a function to create a randomized deck:

createRandomDeck :: forall e. Int -> Eff (random :: RANDOM | e) Deck
createRandomDeck n = shuffle (createDeck n) >>= traverse shuffle

Here, createDeck is the math-y function that returns the same deck always. We call shuffle on it (creating a shuffled deck in an effectful context), and then "bind" that to traverse shuffle. This is possibly the most complicated part of this code. Let's look at traverse:

class (Functor t, Foldable t) <= Traversable t where
  traverse :: forall a b m. Applicative m => (a -> m b) -> t a -> m (t b)
  sequence :: forall a m. Applicative m => t (m a) -> m (t a)

So traverse takes an effectful computation (m is the effect), applies to a traversable container (t is the container) and returns an effectful container of (non-effectful) results. (Huh?) Here t is Array and m is Eff (random :: RANDOM), so this specializes to

traverse :: forall a b. (a -> Eff (random :: RANDOM) b) ->
                        Array a ->
                        Eff (random :: RANDOM) (Array b)

so that traverse shuffle takes an array of "shufflables", shuffles each array element individually, and returns the result in an effectful context. And now we're ready to write our (effectful) function that generates an initial State for the game:

initialState :: forall e. Int -> Eff (random :: RANDOM | e) State
initialState n = do
  cards <- createRandomDeck n
  return { cards : cards, selected : NoCards }

Now that we've gotten actions and state taken care of, it's time to think about how to update the state in response to actions. Here's there's only a single action, which makes it pretty easy:

update :: Action -> State -> EffModel State Action (random  :: RANDOM)
update (Click i) state = { state: cardClicked i state, effects: [] }

The only action is Click i, and so we update the state using the cardClicked i function, which we haven't written yet:

cardClicked :: CardIndex -> State -> State
cardClicked i state = state { selected = toggle state.selected }
  where
    toggle NoCards                      = OneCard i       -- select card i
    toggle (OneCard s1)     | i == s1   = NoCards         -- unselect
                            | otherwise = TwoCards s1 i   -- select second card
    toggle (TwoCards s1 s2) | i == s1   = OneCard s2      -- unselect s1
                            | i == s2   = OneCard s1      -- unselect s2
                            | otherwise = TwoCards s1 s2  -- no op

This is a lot of lines, but conceptually it shouldn't be too hard. When you click on a card you're never changing the deck itself, only the SelectedCards property. If no cards are currently selected, we want to select the clicked card. If one card is selected and we click that card, we want to unselect it; if we click a different card we want to select the second card as well. And if two cards are already selected, we either unselect one (if we clicked it), or do nothing (if it's a different card).

At last, we're ready to create the view. As usual, we have a lot of logic that involves the "index" of an array element, so we'll need a helper function to map across elements and their indexes:

mapWithIndex :: forall a b. (a -> Int -> b) -> Array a -> Array b
mapWithIndex f xs = map (uncurry f) $ zip xs (0 .. (length xs - 1))

And since we'll want to add a special CSS class to the common image between two selected cards, we'll also need a helper function to find it:

commonImage :: Deck -> CardIndex -> CardIndex -> Image
commonImage cards i j = head $ do
  image1 <- cards `unsafeIndex` i
  image2 <- cards `unsafeIndex` j
  guard $ image1 == image2
  return image1

This is possibly a bad way to write it, since it will crash if the cards had no common image, but if our math is correct that will never happen. Right?

So then the view:

view :: State -> Html Action
view state = div [] cardsHtml
  where
    cardsHtml = mapWithIndex (renderCard state.selected correctImage) state.cards
    correctImage = case state.selected of
      TwoCards i j -> Just (commonImage state.cards i j)
      _            -> Nothing

Curiously, right when I finished the first version of this project, pux 1.0.0 dropped, with a large number of breaking changes, so I had to redo a lot of stuff. In particular, there is a sugary hypertext DSL, but it doesn't cover every use case, so I decided not to use it, since the code looked weird with a mix of DSL and not DSL.

Here the div function takes an array of attrbutes (here, empty) and an array of child Html elements, which are the outcome of the call to mapWithIndex. The function we pass to mapWithIndex is renderCard state.selected correctImage. In order to figure out the correct image, we need to look at multiple cards, so we have to do it at this stage (each call to renderCard will only know about the card it's given). If 0 or 1 cards are selected, there is no correctImage, so it's an option type. We also have to pass down state.selected so that we can add a CSS class to the selected cards.

This leads to a renderCard that looks like

renderCard :: SelectedCards -> Maybe Image -> Card -> CardIndex -> Html Action
renderCard selectedCards correctImage card i =
  div [ className cardClass, onClick cardClick ] cardHtml
  where
    isSelected = case selectedCards of
      NoCards        -> false
      OneCard s      -> s == i
      TwoCards s1 s2 -> s1 == i || s2 == i
    cardClass = if isSelected then "card selected" else "card"
    cardClick = const (Click i)
    cardHtml = map (renderImage isSelected correctImage) card

Because we're passing it to mapWithIndex, it's a function of both Card and CardIndex (after we curry the SelectedCards and the correct image). Again we create a div with an array of attributes (a class, and a click handler) and an array of Html children.

Hopefully the class logic is straightforward. cardClick should be a function that takes a click event and returns an Action, but here we don't actually need any information from the click event, so we just use const.

Finally, the cardHtml is just the result of mapping renderImage over the card (which is an array of images). It needs to know isSelected and correctImage, because we want to add the flashing "correct" class only the the correct image on the selected cards.

At long last we can write renderImage:

renderImage :: Boolean -> Maybe Image -> Image -> Html Action
renderImage isSelected correctImage image =
  img [ src url, alt altText, className imageClass ]
  where
    url = imageUrl image
    altText = show image
    imageClass = if isCorrectImage then "image correct" else "image"
    isCorrectImage = case correctImage of
      Just correct -> isSelected && image == correct
      _            -> false

Because an <img> tag has no children, the img function only takes an array of attributes, which should be pretty straighforward.

That's the whole application, now we just need to wire it all together:

main = do
  state <- initialState 7  -- you could change this, if you have enough images
  app <- start {           -- but it *must* be a prime number
    initialState: state
  , update: update
  , view: view
  , inputs: []
  }

  renderToDOM "#app" app.html

The only non-ordinary thing here is that because initialState is an effectful function, we need to <- the state out of it.

Anyway, that's about it. (There is also some CSS voodoo to make matching images pulse) and to draw borders around selected cards, but you're not interested in CSS voodoo, are you?)

You can check it out at

https://joelgrus.com/experiments/pux-it/

although the bundle of all the code is almost 1MB. :( Or fork it and make it your own. :)

Trump Tweets on a Globe (aka Fun with d3, socket.io, and the Twitter API)

2016-02-27T09:00:00+01:00

When I worked at Farecast we had a giant TV in the lobby, and up on that TV was an image of a globe, and on that globe were animated paths that (as far as you know) corresponded to flight searches that people were doing on the site. It was a big hit with visitors to the office, and I've always wanted to recreate it myself.

These days I don't have access to flight search data, but I do have access to (a subset of) the Twitter firehose, some subset of which is tweets tagged with geographic data. Hence today's stupid project:

Trump Tweets on a Globe

It streams tweets about Donald Trump (or whatever other topic you want), keeps only the ones that contain geo information, and then plots them on a globe.

(all code on GitHub, same as always)

And if anyone from the Trump campaign wants to hire me to build stupid junk like this for them, I'm available!

The Client

There is a saying (that I made up):

Good data scientists copy from the d3 gallery; great data scientists steal from the d3 gallery.

Accordingly, most of the visualization guts are shamelessly cribbed from Mike Bostock's World Tour visualization. (Which infected my code with the GPL, which I guess is the point of the GPL.)

His version has a list of countries that it simply rotates through in order, while ours needs to react to Tweets as they arrive.

My first attempt was to simply stick all of the Twitter logic in the client and browserify it together, but that didn't work for cross-origin reasons, so I ended up having to stick the Twitter piece on the server.

Then, because it seemed like the sensible thing to do, I decided that the server should maintain a single Twitter client and use socket.io to send tweets to the browser. (I have never really used socket.io before, so it's possible/likely I'm doing so in a really stupid way.)

The relevant part of the original visualization is (comments added by me):

(function transition() {
  d3.transition()
      // Start a 1.25 second transition.
      .duration(1250)
      // Increment `i` and set the title to the relevant country name.
      .each("start", function() {
        title.text(countries[i = (i + 1) % n].name);
      })
      // Rotate the globe to put that country in front.
      .tween("rotate", function() {
        var p = d3.geo.centroid(countries[i]),
            r = d3.interpolate(projection.rotate(), [-p[0], -p[1]]);
        return function(t) {
          projection.rotate(r(t));
          // Clear the canvas
          c.clearRect(0, 0, width, height);
          // Fill the land gray
          c.fillStyle = "#ccc", c.beginPath(), path(land), c.fill();
          // Fill the selected country red
          c.fillStyle = "#f00", c.beginPath(), path(countries[i]), c.fill();
          // Draw the country borders in white
          c.strokeStyle = "#fff", c.lineWidth = .5, c.beginPath(), path(borders), c.stroke();
          // Circle the globe in black
          c.strokeStyle = "#000", c.lineWidth = 2, c.beginPath(), path(globe), c.stroke();
        };
      })
    // Start a new transition that does nothing and then recursively calls
    // transition() to move to the next country.
    .transition()
      .each("end", transition);
})();

How should we modify this? Well, first, we need to open a socketio connection to receive tweets.

var socket = io({ "force new connection" : true });

And then instead of having our transitions triggered by an IIFE that calls itself recursively, we invoke a transition whenever we receive a 'tweet' message over the socket:

socket.on('tweet', function(tweet) {
  console.log(tweet);
  d3.transition()
    .duration(1250)
    .each("start", function() {
      title.html(makeHTML(tweet));
    })
    .tween("rotate", function() {
      var p = tweet.latLong,
          r = d3.interpolate(projection.rotate(), [-p[0], -p[1]]);
      return function(t) {
        projection.rotate(r(t));
        c.clearRect(0, 0, width, height);
        c.fillStyle = "#ccc", c.beginPath(), path(land), c.fill();
        c.strokeStyle = "#fff", c.lineWidth = .5, c.beginPath(), path(borders), c.stroke();
        c.strokeStyle = "#000", c.lineWidth = 2, c.beginPath(), path(globe), c.stroke();
        var center = projection(p);
        c.strokeStyle = "#000", c.fillStyle = "#f00", c.beginPath(), c.arc(center[0], center[1], 5, 0, 2 * Math.PI, false), c.lineWidth = 2, c.fill(), c.stroke();
      };
    });
});

The major differences here are

instead of using title.text() to set the country name, we use title.html() to create more complex HTML (see below)
instead of getting the "front and center" coordinates from the array of countries, we get them from the latLong field of the tweet message.
we add an extra drawing to the canvas -- a filled-in circle at the latLong coordinates, after we use projection to convert them into the canvas coordinate system
there's no recursive call at the end, since the transition will get invoked again every time there's socket message.

Otherwise it's the same! (I had to add some more CSS as well.)

We still need to write the function that makes HTML:

function makeHTML(tweet) {
  return [
    '<div class="user">',
    '<a href="https://twitter.com/', tweet.user, '" target="_blank">', '@', tweet.user, '</a>',
    '<h1>', tweet.placeName, '</h1>',
    '<div class="tweet">',
    '<a href="https://twitter.com/', tweet.user, '/status/', tweet.id, '" target="_blank">',
    tweet.text, '</a>', '</div>'
  ].join('');
}

It's really ugly (my kingdom for a templating engine!) but all it's doing is showing the Twitter user name (with a link), the place name of the specified location, and the text of the tweet (With a link).

The Server

OK, now we're ready to deal with the server. It will do a few things:

Serve up our index.html page.
Subscribe to some topic on the Twitter streaming API.
Massage the returned tweets and publish them to socketio.

As always, you'll need to get some Twitter credentials and set them up in a credentials.js that looks like

module.exports = {
  consumer_key: "...",
  consumer_secret: "...",
  access_token_key: "...",
  access_token_secret: "..."
};

And then our server is actually not that complicated. Let's start out with all the express boilerplate:

var express = require('express'),
    app = express(),
    http = require('http').Server(app);

// Serve index.html at the root.
app.get('/', function(req, res){
  res.sendFile(__dirname + '/index.html');
});

// Serve static files in the public directory.
app.use(express.static('public'));

// Run on port 3000.
http.listen(3000, function() {
  console.log('listening on 3000');
});

Now, because we're working in Javascript, we've got async stuff everywhere. In particular, we'll need to give our Twitter client a callback function that triggers whenever a new tweet is received. In my first version that callback function published to socket.io directly. For reasons I can't precisely explain to you, that struck me as too tight of a coupling, so I introduced an intermediate TweetEmitter:

// This is all OO-boilerplate-cruft to make my own event emitter.
// https://nodejs.org/api/events.html
var EventEmitter = require('events'),
    util = require('util');

function TweetEmitter() {
  EventEmitter.call(this);
}
util.inherits(TweetEmitter, EventEmitter);

var tweetEmitter = new TweetEmitter();

There's pretty much nothing going on here. The Twitter client will send its tweets to the TweetEmitter, and the socketio socket will grab tweets from the TweetEmitter and that way they don't have to know about each other, which felt like the right design.

At this point it's easy to hook the TweetEmitter up to socket.io:

var io = require('socket.io')(http);

tweetEmitter.on('tweet', function(tweet) {
  console.log(tweet);
  io.emit('tweet', tweet);
});

Finally we need to set up the Twitter client.

There is a slight complication here, in that there are several ways in which a tweet might specify its geography. A tweet has a geo field, which I believe is deprecated. It has a coordinates field, which is rarely populated. And it has a place field, which (is sometimes populated and if so) contains a rich object. For our purposes the most interesting properties of place are

full_name : we'd like to display it on our globe, and
bounding_box.coordinates : an array of arrays of (longitude, latitude) pairs.

(I assume it's an array of arrays because the "bounding box" might comprise multiple polygons? I'm not sure.)

Anyway, we just want to put a dot at a single location, which we'll accomplish by averaging all of the coordinate pairs:

// a helper function to average coordinate pairs
function average(coordinates) {
  var n = 0, lon = 0.0, lat = 0.0;
  coordinates.forEach(function(latLongs) {
    latLongs.forEach(function(latLong) {
      lon += latLong[0];
      lat += latLong[1];
      n += 1;
    })
  });
  return [lon / n, lat / n];
}

(Yes, I am aware that I called them latLongs even though the longitude is first. I always do that, it's a character flaw.)

Next we setup Twitter and get our search term from the command line args if appropriate (or just use trump as the default):

var Twitter = require('twitter'),
    credentials = require('./credentials.js'),
    client = new Twitter(credentials);

var query = process.argv[2] || 'trump';

And finally we start listening to the streaming API:

client.stream('statuses/filter', {track: query}, function(stream) {
  // Every time we receive a tweet...
  stream.on('data', function(tweet) {
    // ... that has the `place` field populated ...
    if (tweet.place) {
      // ... extract only the fields needed by the client ...
      var tweetSmall = {
        id: tweet.id_str,
        user: tweet.user.screen_name,
        text: tweet.text,
        placeName: tweet.place.full_name,
        latLong: average(tweet.place.bounding_box.coordinates),
      }
      // ... and notify the tweetEmitter.
      tweetEmitter.emit('tweet', tweetSmall);
    }
  });
});

And that's all of it. If you've read this far, you can (maybe) see it up and running at

http://54.174.99.38:3000/

That's a tiny ec2 nano instance that's trying to socket.io with everyone who goes to that URL, so it will probably crash eventually, at which point you'll need to run it yourself locally.

So that's it. For once no Haskell, no PureScript, not even any Python, just clean old node and JavaScript.

Possible Improvements

As I mentioned above, this approach probably doesn't scale to very many users before the backend just gets overwhelmed. So one obvious improvement would be to scale it up. If I wanted to do that, my first attempt would probably be to have one dedicated process publishing the tweets to a (private) socket, and a bunch of express processes behind a load balancer, each of them pulling tweets from the (private) socket and publishing them to their clients. Conceptually that shouldn't be too hard, but it seems like more devops work and/or AWS bill than I want to devote to a toy project at this point.

One person I showed it to wanted the tweets throttled, so that the globe updated more slowly. Personally, I like it the way it is. One approach would be to throttle things on the server side (only publish to the socket after a certain interval has passed), another would be on the client side (throw away messages until some interval has passed). In the second case you could even give the user a slider to choose the update speed.

Another person wanted a query box to select their own search terms. Unfortunately, that doesn't fit into this model -- the Twitter client needs a separate streaming connection to Twitter for each search term, and it defeats the point of sharing the single Twitter connection among all the connected clients.

It also might be fun to leave the dots and tweets on the globe for a while and have them slowly fade away. That seems like a lot more work. (Especially since the tweets aren't technically "on the globe", they're in a fixed spot.)

So anyway, check it out, check out the code, roll your own version, and if anyone from the Trump campaign is reading this, I wasn't kidding, call me!

Got any other ideas to improve it? Let me know!

Building a Stupid Data Product, Part 3: The Single-Page App (PureScript)

2016-02-15T09:00:00+01:00

(part 1, part 2)

OK, so last time we built a web service that responds to GET requests with random (bogus) elementary school science questions. In this third (and last) installment, we'll create a single-page quiz app that consumes the service.

Here's a rough stab at a wireframe:

On the left will be the quiz questions. On the right will be the score, a "new game" button, and self-aggrandizing stuff like links to my Twitter. When a question appears, its answers will all be gray. Once you click on one of the answers, it will get a bold border, and we'll color the correct answer green and the incorrect ones red.

NOW, you know me, I never met a fancy PureScript framework I didn't like. In this case it's the brand new purescript-pux, which is a FRP interface to React. (I'm not that into React, but the framework insulates you away from using it.)

(Code, as always, is on GitHub.)

Types

As is almost always the case when I work with Haskell or PureScript, I start with the types:

newtype Question = Question {
  questionText  :: String,
  answers       :: Array Answer,
  correctAnswer :: AnswerId,
  chosenAnswer  :: Maybe AnswerId  -- which answer the player clicked on
}

type QuestionId = Int
type Answer     = String
type AnswerId   = Int

A few things here. The Question type is basically a record that looks like the JSON responses we'll get from the service. In addition, since our wireframe called for different formatting (i.e. different css classes) for selected / correct / wrong answers, I added a chosenAnswer field that's either Nothing (if the question hasn't been answered yet) or Just answerId (if the quizzee clicked on the answerId answer). And in order to make it an instance of the IsForeign typeclass (which makes it easy to convert the JSON responses from AJAX calls into typed objects) we need to wrap it in a newtype.

I also made type aliases for QuestionId, Answer, and AnswerId just to make our code nicer and more descriptive.

Anyway, like I said, we can now define an IsForeign instance for Question:

instance questionIsForeign :: IsForeign Question where
  read value = do
    questionText  <- readProp "questionText"  value
    answers       <- readProp "answers"       value
    correctAnswer <- readProp "correctAnswer" value
    return $ Question {
      questionText : questionText,
      answers : answers,
      correctAnswer : correctAnswer,
      chosenAnswer : Nothing
    }

Here read tells us how to turn the JSON object value into a Question. The type of read is

read :: Foreign -> Either ForeignError Question

You can think of this (if you like) as a function that takes a Foreign object (i.e. some untyped JSON), and returns a Question object in a Either ForeignError context. In other words, it describes a computation that might return a Question but might fail with some kind of ForeignError. In that context, the line

questionText  <- readProp "questionText" value

tries to get the value of the questionText field as a String. If it succeeds, it goes on to the next line. If it fails, the whole computation fails with the corresponding ForeignError.

If we successfully read these fields, we can create a Question object (with, in addition, its chosenAnswer field set to Nothing), and then use return to stick it in the Either ForeignError context.

FRP

Similar to our previous stabs at FRP, our app will consist of

State, which keeps track of the state of the app
Actions, which update the state (in a pure way) and kick off side effects
View, which turns the state into a VirtualDOM that can be rendered by React

State

Let's deal with the state first:

type State = {
  score              :: Int,
  questions          :: Array Question,
  waitingForQuestion :: Boolean -- are we waiting for an AJAX call to return?
}

Our state consists simply of a score (the number of questions you've answered correctly), an array of questions, and a flag that indicates whether the app is waiting for a question. (The initial version didn't have this flag, which caused a bug where if you clicked the "New Game" button a lot of times quickly, it would fire off a lot of AJAX requests and then add all the questions when they eventually returned. We just want to add one question at a time.)

Actions

Now we can deal with the actions, of which there are three:

data Action =
    NewGame                          -- start a new game
  | ClickAnswer QuestionId AnswerId  -- click on an answer
  | QuestionReceived Question        -- receive an AJAX response with a question

NewGame is the action to take when someone clicks on the "New Game" button. ClickAnswer is the action to take when someone clicks on one of the answers. Its payload contains the id of the question and the id of the answer, since we'll need both of those to update the state. And QuestionReceived is the action to take when the app receives the result of an (asynchronous) call to the random question service. Its payload is the received Question (as a typed PureScript object).

Next we'll write the code that updates the state in response to each of these actions. For NewGame we'll just replace the state with:

initialState :: State
initialState = { score: 0, questions: [], waitingForQuestion: true }

For the QuestionReceived action, we just append the question to state.questions. (That's what snoc does. It's cons backward.) The only subtlety involves the waitingForQuestion flag; if it's false then we don't add the question; if it's true then we set it to false. The flag ensures that we add at most one question per "intra-game question request".

appendQuestion :: Question -> State -> State
appendQuestion question state =
  if state.waitingForQuestion
  then state { questions = snoc state.questions question,
               waitingForQuestion = false }
  else state

And we need a function that updates the state after a ClickAnswer action. It needs to know the QuestionId and AnswerId that were clicked on.

answerClicked :: QuestionId -> AnswerId -> State -> State
answerClicked questionId answerId state =
  { score : newScore, questions: newQuestions, waitingForQuestion: true }
  where
    q = case state.questions `unsafeIndex` questionId of Question q' -> q'
    newScore =
      if q.correctAnswer == answerId then state.score + 1 else state.score
    answeredQuestion = Question $ q { chosenAnswer = Just answerId }
    newQuestions =
      fromJust $ updateAt questionId answeredQuestion state.questions

This one is a little more complicated. First we use unsafeIndex to pull the clicked Question out of state.questions, along with pattern-matching to pull the question record out of the newtype. We compute the new score by adding 1 to the current if the clicked answerId was the correctAnswer. Then we update the question at questionId by setting its chosenAnswer property.

So, at this point we have Actions and we have functions that update the state in response to each action type. Now we need to glue them together:

-- How to update the state (and perform effects) for each action type.
update :: forall eff. Update (ajax    :: AJAX,
                              err     :: EXCEPTION,
                              console :: CONSOLE    | eff) State Action
update action state input =
  case action of
    NewGame ->
      { state: initialState
      , effects: [ requestQuestion ] }
    ClickAnswer questionId answerId ->
      { state: answerClicked questionId answerId state
      , effects: [ requestQuestion ]
      }
    QuestionReceived question ->
      { state: appendQuestion question state
      , effects: [] }

The Update function takes in an Action, the state, and input (which is part of the plumbing for asynchronous state changes), and returns a record with the new state and an array of side-effects.

Since we have three Action types, we use pattern matching to handle the three different cases. For NewGame, we reset the initial state and make an AJAX request for a new question. For ClickAnswer we call our answerClicked function to get a new state and also make an AJAX request for a new question. And for QuestionReceived we use our appendQuestion function to update the state.

You're probably wondering what requestQuestion is. It's pretty much boilerplate around making an AJAX call, and to be honest I don't really understand it well (I mostly copied it from the documentation).

  where
    requestQuestion =
      launchAff $ do
        res <- get questionServiceUrl
        let question = readJSON res.response :: F Question
        liftEff $ case question of
          (Left err) -> log "Error parsing JSON!"
          (Right question) -> S.send input (singleton (QuestionReceived question))

Here launchAff takes a value in an "asynchronous computation effect context", runs it synchronously, and ignores the value. (This is fine, since we don't need the result of the computation, we just want it to emit a signal when it finishes (or log an error if it fails)).

The asynchronous computation makes an GET request to the questionServiceUrl, uses readJSON to convert the response into a Question object (in the F context, which is a type synonym for Either ForeignError) and then either logs the error (if the conversion fails) or sends off a QuestionReceived action (if the conversion succeeds).

(Because both the log and S.send effects operate in the synchronous effect context Eff, we have to "lift" them into the asynchronous effect context of requestQuestion.)

View

Now it's time to produce the view. At a high level our goal is

view :: State -> VirtualDOM

after which React will take care of rendering the VirtualDOM.

VirtualDOM has a hyperscript DSL that allows you to write stuff like

-- This is an illustrative example, not part of our code:
view :: State -> VirtualDOM
view state = div $ do
  p $ text ("Counter: " ++ show state.counter)
  p $ do
    button ! onClick (send Increment) $ text "Increment"
    button ! onClick (send Decrement) $ text "Decrement"

The place where I got stuck was on how to create arbitrarily many elements by map-ing over an array. (None of the examples do that.) Eventually I noticed that VirtualDOM has a Monoid instance, which means we can use

foldMap :: forall a m. (Monoid m) => (a -> m) -> Array a -> m

with the types in this particular case specialized as

foldMap :: forall a. (a -> VirtualDOM) -> Array a -> VirtualDOM

which means we could do stuff like

-- This is an illustrative example, not part of our code:
showTenUsers :: State -> VirtualDOM
showTenUsers state =
  foldMap (\user -> p $ text user.name) (take 10 state.users)

Now because we're using array indexes as QuestionId and AnswerId, we really want a variant that allows the generated VirtualDOM elements to also depend on the index, which I'll call foldMapWithIndex:

foldMapWithIndex :: forall a. (a -> Int -> VirtualDOM) -> Array a -> VirtualDOM
foldMapWithIndex f xs = foldMap (uncurry f) pairs
  where pairs = zip xs (0 .. (length xs - 1))

As desired, it takes a function that generates a VirtualDOM from an element and its index, uncurrys it (i.e. converts it from a function of two arguments to a function of one Tuple argument), and foldMaps it over the pairs (element, index).

Now we're ready to write our view. We'll start at the highest level and work our way down:

view :: State -> VirtualDOM
view state = div $ do
  div ! className "sidebar" $ do
    p ! className "score" $ text ("Score: " ++ show state.score)
    button ! onClick (send NewGame) $ text "New Game"
    p ! className "twitter" $
      a ! href "http://twitter.com/joelgrus" $ text "@joelgrus"
    p ! className "github" $
      a ! href "https://github.com/joelgrus/science-questions" $
        img ! src "octocat.png"
  foldMapWithIndex renderQuestion state.questions

OK. So our view produces a div with two VirtualDOM children.

The first child is a "sidebar". We'll use css to float it off to the right. It contains a p showing the current score, a "New Game" button, a link to my Twitter account, and a link to the GitHub repo for this project.

The only two interesting parts are

the "score", which looks into the state to find the score, and
the "new game" button, which uses send to trigger a NewGame action

The second child is the output of a call to foldMapWithIndex, which (as we described above) feeds each element of state.questions and its index to the renderQuestion function.

This tells us that we must have

renderQuestion :: Question -> Int -> VirtualDOM

Or, since QuestionId is a type alias for Int, we can use the more descriptive

renderQuestion :: Question -> QuestionId -> VirtualDOM
renderQuestion (Question q) questionId = div $ do
  p ! className "question" $ text $ questionNumber ++ ". " ++ q.questionText
  foldMapWithIndex (renderAnswer (Question q) questionId) q.answers
  where
    questionNumber = show (questionId + 1)

Here we render a question as a p with the question number and question text, followed by another foldMapWithIndex across the question's answers.

Now we need to write renderAnswer. Notice our usage: we passed it a Question and a QuestionId, after which (since we're using foldMapWithIndex) we need to end up with a function that takes an Answer and an AnswerId:

renderAnswer :: Question -> QuestionId -> Answer -> AnswerId -> VirtualDOM
renderAnswer (Question q) questionId answer answerId =
  div ! className classes ! clickHandlerIfUnanswered $ text answer
  where
    isAnswered = isJust q.chosenAnswer
    isChosen   = isAnswered && answerId == fromJust q.chosenAnswer
    isCorrect  = answerId == q.correctAnswer

    classes = joinWith " " $ map snd $ filter fst [
      Tuple true                          "answer",
      Tuple isChosen                      "chosen",
      Tuple (isAnswered && isCorrect)     "correct",
      Tuple (isAnswered && not isCorrect) "wrong"
    ]

    clickHandlerIfUnanswered =
      if isAnswered
      then Attrs [] [] -- no-op "attribute"
      else onClick (send $ ClickAnswer questionId answerId)

How on earth is rendering a measly little answer so involved? Well, two reasons.

Our original wireframe involved a lot of different state-dependent styles for answers, which means that we need to assign state-dependent classes in a slightly complicated way.
The main activity of the game involves clicking on answers, which means that we need to set up click handlers.

The hyperscript part is quite simple:

  div ! className classes ! clickHandlerIfUnanswered $ text answer

It's a div, with some classes attached to it, with something else that might be a click handler attached to it, and containing the text of the answer. Simple.

The classes are determined by three boolean values:

isAnswered -- has this question been answered (at all)
isChosen -- did the quizzee click this answer?
isCorrect -- is this the correct answer?

The code for generating the classes is possibly too clever (although all my alternative versions with lots of if and then were really ugly), but basically it sets up an array of pairs (boolean value, class name), throws out the pairs where the first element is false, and joins together the class names that are left.

For the click handler, we only want answers clickable if they belong to a question that hasn't been answered yet. The simplest way I could come up with to do that was the if-then statement that produced either a click handler or an "empty attribute" depending on the value of isAnswered.

I think that's pretty much it, other than the main boilerplate:

main = renderToDOM "#app" =<< app
  { state: initialState
  , update: update
  , view: view
  , inputs: []
  }

The Result

You can see it up and running here. (Fear my mad CSS / design skills.)

The front-end should be totally robust, but the back-end question service is running on an EC2 nano instance, so try to be gentle. If it dies for some reason, this is what it looks like (or run it yourself locally!):

The Verdict

I really enjoyed working with purescript-pux, it's probably my favorite of the PureScript frameworks I've tried. It's pretty much brand new (I think it was announced a couple of weeks ago), so there's not a ton of help, but the docs are pretty good and I managed to figure everything out mostly. Why do I like it?

It has pretty much the same model as cycle.js, which I've been playing with a lot recently, and "feels like" the right way to write apps.
From my (newbie) perspective, it involves a good bit less mystery than (e.g.) purescript-thermite or purescript-halogen, neither of which I really felt like I was understanding as I used them.

So check it out, and thanks to @alexmingoia for creating it.

Anyway, that's the end of our end-to-end stupid data product. Possibly you learned something, and possibly you'll go off and build your own stupid data product (or maybe even a not-stupid one). If you do, let me know about it!

Building a Stupid Data Product, Part 2: The Web Service (Haskell)

2016-02-15T08:00:00+01:00

(part 1, part 3)

Last time we collected and processed the data for generating stupid fake elementary school science questions and answers. The important parts to remember are

we generated two files questions.json and answers.json containing transition dictionaries mapping each word to an array / list of possible following words, and
we used sentinel tokens __START__ and __STOP__ to indicate the beginning and end of sentences.

In this post we'll use a Haskell library called servant to build a web service that generates and returns random questions. If you know Haskell, I'm sure you can find fault with the way I did things, but if you don't know Haskell you might find it educational and/or mind-expanding.

(Code, as always, is on GitHub.)

We'll do this in four steps:

define some types
write code that can generate a random question, given an (abstract) GetNextToken function
implement a (concrete) GetNextToken based on our transitions
create an API that serves up the random questions

The Types

Let's start with the types. We need to define the Question that our web service will return. In our API, a question will have a questionText, a list/array of answers (which are just strings), and an integer indicating the index of the correctAnswer. Pretty simple:

data Question = Question
  { questionText  :: String
  , answers       :: [Answer]
  , correctAnswer :: Int
  } deriving (Eq, Show)

type Answer = String

$(deriveJSON defaultOptions ''Question)

The last line is (I believe) some template Haskell voodoo that makes it so our service knows how to serialize a Question to JSON (since we can't send Haskell objects over the wire). I don't understand it, I just copied it from the docs.

Now we need to define a type for our tokens. One of the benefits of working in a nicely-typed language is that we don't have to use "sentinel values", we can use our type system for that:

data Token = Start | Stop | Word String deriving (Eq, Ord)

So a token is either Start, Stop, or a Word with an associated String value. The deriving (Eq, Ord) just makes it so that we can test two tokens for equality and inequalities.

Since our tokens will come from deserializing JSON, we'll also need a Read instance, which indicates how to parse text into Token objects:

instance Read Token where
  readsPrec _ "__START__" = [(Start,  "")]
  readsPrec _ "__STOP__"  = [(Stop,   "")]
  readsPrec _ w           = [(Word w, "")]

Don't get hung up on the details, it does exactly what you'd expect it to do. (If you do get hung up on the details, read the docs.)

We also want to define a type alias

type GetNextToken = Token -> IO Token

that represents a function that takes a Token and returns an IO Token. If you are not a Haskell person, you are at this point wondering

Why does it not just return a Token?
What the hell is an IO Token?

For the first, Haskell is a pure functional language. This means that if you tried

type GetNextTokenBad = Token -> Token

any instance of GetNextTokenBad would have to always return the same value for the same input. In particular, it wouldn't be able to choose the next token randomly. If we want side-effects like randomness (or printing things, or reading from files), we need to do computations in the IO context. So when you see

type GetNextToken = Token -> IO Token

you can understand that as a function that takes a token, does something side-effectful, and returns a new token in the IO context. In particular, this function doesn't need to return the same value for the same inputs, but also you can only use it in a context that allows side effects. More on that in a bit.

Generating Random Questions

Now we're ready to write the code for generating a sentence. This is where things start to get a little complicated. We'll break it into two parts. First, given a starting Token and a GetNextToken function, we want to generate a list of Tokens in the IO context:

tokensFrom :: Token -> GetNextToken -> IO [Token]
tokensFrom startToken getNext = do
  nextToken <- getNext startToken   -- nextToken :: Token
  case nextToken of
    Stop  -> return []
    token -> liftA2 (:) (pure token) (tokensFrom token getNext)

This shouldn't be hard conceptually, it's just recursion:

tokensFrom takes a start Token and a GetNextToken function
it calls the GetNextToken function on the starting Token
if nextToken is Stop, the result is an empty list;
otherwise, the result is the list whose first element is nextToken, and whose subsequent elements are the results of tokensFrom nextToken.

In reality, it's complicated because of the need to do things in an effectful context. The do is sugar for working in the IO context. In particular, it allows us to pull the Token value out of the result of a GetNextToken call. That is, while getNext returns an IO Token, as long as we're inside the do block for an IO context, we can use <- to "get the Token out."

If we find Stop, the result is return []. Notably, this is not the return you might know from other languages. Here this is

return :: a -> IO a

which sticks a value (in this case the empty list) into an IO context. So, since [] is a [Token], return [] is an IO [Token].

The last line is even uglier. (:) is the "cons" operator that takes a head and a tail and produces a list:

(:) :: a -> [a] -> [a]

Here nextToken is a Token, but the recursive call to tokensFrom produces an IO [Token], so the types don't match up. We've already seen that we can shove values into an IO context, so we could get by if we had something like

-- | not a real operator
(:???) :: IO a -> IO [a] -> IO [a]

We can get there with liftA2, which (specialized for IO) looks like

liftA2 :: (a -> b -> c) -> IO a -> IO b -> IO c

That is, it "lifts" a function of two arguments into an IO context. If you work through the types, you get:

liftA2 (:) :: IO a -> IO [a] -> IO [a]

which is exactly what we want.

[Why did I use pure instead of return to stick nextToken into the IO context? I'm not sure, exactly. In this case they're the same thing. In the previous instance I was using IO as a Monad, so I used return; here I'm using it as an Applicative, so I used pure. That's not a good explanation, and it's probably not even a good reason. I don't care. (I was also trying not to say "monad" in this post, but I guess I failed.)]

Next we want to turn a list of Tokens into a String:

smartJoin :: [Token] -> String
smartJoin = dropWhile (== ' ') . concat . addSeparators
  where
    addSeparators = concatMap addSeparator
    addSeparator token = case token of
      Word w | w `elem` ["?", ",", "."] -> ["",  w]
      Word w                            -> [" ", w]
      _                                 -> []

The first thing we do is addSeparators, which turns each Word into a list [separator, word] and then concatenates the resulting lists. If the Word is punctuation, the separator is an empty string. Otherwise it's a space.

(We should never call smartJoin on a list that includes the Start or Stop tokens, but just in case we add in an empty list, which is the same as ignoring the token.)

So, for instance, if you were to call

addSeparators [Word "What", Word "is", Word "love", Word "?"]

you would get

[" ", "What", " ", "is", " ", "love", "", "?"]

We then call concat on that to concatenate all the strings

" What is love?"

and dropWhile (== ' ') to get rid of the leading spaces. (I know, sort of clunky.)

Now we're ready to implement our sentence generator:

generate :: GetNextToken -> IO String
generate = fmap smartJoin . tokensFrom Start

To be a jerk, I wrote it in point-free style, it's the same as if I'd done

generate nextToken = fmap smartJoin (tokensFrom Start nextToken)

Here tokensFrom generates IO [Token] (an list of tokens in an effectful context) and fmap lifts smartJoin (which maps [Token] -> String) into the IO context, resulting in our desired IO String.

And finally we can create our Question generator:

randomQuestion :: Int -> GetNextToken -> GetNextToken -> IO Question
randomQuestion numAnswers getNextQuestionToken getNextAnswerToken =
  Question <$> generate getNextQuestionToken
           <*> replicateM numAnswers (generate getNextAnswerToken)
           <*> randomRIO (0, numAnswers - 1)

It takes an Int indicating how many answers the question should have. And it needs two GetNextToken functions, one for generating questionText and the other for generating Answers.

You can think of <$> and <*> as plumbing to lift the Question constructor into the IO context. That's a Haskell-y way of doing (in essence)

-- the constructor is in essence
-- Question :: String -> [Answer] -> Int -> Question

makeEffectfulQuestion :: IO String -> IO [Answer] -> IO Int -> IO Question
makeEffectfulQuestion = liftA3 Question

Here the IO String comes from generate-ing the question, the IO [Answer] comes from using replicateM to generate multiple answers, and the IO Int comes from choosing a random "correct answer".

Using Transitions

Now that we have a way to generate Questions using GetNextToken functions, we have to figure out how create GetNextToken functions from the transition maps we generated last time. We serialized them as JSON, but now we want a typed way to work with them in Haskell:

type Transitions = M.Map Token [Token]

Here Transitions is a Map (like a dictionary) whose keys are Tokens and whose values are lists of Tokens.

However, our serialized map of transitions is a dictionary whose keys are strings and whose values are lists of strings. That means we need to deserialize it and then convert the strings to Tokens:

loadTransitions :: String -> IO Transitions
loadTransitions = fmap (textToTokens . fromJust . decode) . BS.readFile
  where textToTokens = M.map (map read) . M.mapKeys read

Our loadTransitions is another point-free function. It reads a file (which gets us some bytes in an IO context), and then uses fmap to lift the three composed functions into the IO context.

First, decode Maybe-deserializes the bytes into a map (with text keys and values). After that, fromJust assumes the deserialization succeeded and pulls the map out of the Maybe. Finally, textToTokens converts the text-texts map into a Token-Tokens map.

(The fromJust isn't a "safe" way to do things (usually we'd want to check that decode doesn't return Nothing and deal with that somehow), but because we generated the JSON ourselves, we know it's valid.)

How does textToTokens work? First, it calls M.mapKeys read, which returns the new Map that results from applying read to each of the input Map's keys. So it returns a map whose keys are Tokens but whose values are still lists of text. And then we feed it into M.Map (map read), which returns the Map that results from calling map read on each of the input Map's values. Those values are lists of text, so map read converts each one to a list of Tokens. At the end of the process we have a M.Map Token [Token] as required.

Now we're ready to actually load the data:

questionTransitions :: IO Transitions
questionTransitions = loadTransitions "questions.json"

answerTransitions :: IO Transitions
answerTransitions = loadTransitions "answers.json"

Next, remember that the abstraction we used was

type GetNextToken = Token -> IO Token

so we simply need to implement a function like this that uses our Transitions. First we write a function to pick a random element of a (nonempty) list. We get a random Int (in an IO context, of course) and use it to index into the list:

-- will crash if the input is an empty list
pick :: [a] -> IO a
pick xs = do
  idx <- randomRIO (0, length xs - 1) -- choose a random index
  return (xs !! idx)                  -- return that element of the list

And then our implementation is easy, we just create a function that takes as input a Transitions object and returns the corresponding GetNextToken function:

randomNextToken :: Transitions -> GetNextToken
randomNextToken transitions token =
  case M.lookup token transitions of
    Just tokens -> pick tokens
    _           -> return Stop  -- this shouldn't happen, but let's be safe

If you are confused about why we define it as randomNextToken transitions token, substitute in the definition of GetNextToken:

randomNextToken :: Transitions -> Token -> IO Token

Once it's applied to a Transitions object, what's left is a function that looks up a token in the Transitions map and pick one of the following tokens at random.

The API

Finally, we're ready to create the actual web service. To start with, we define our API:

type API = "question" :> Get '[JSON] Question

It has a single endpoint "question", which responds to HTTP GET requests and returns a Question serialized into JSON.

My first attempt at implementing this turned out to be really slow. After poking around at a lot of stuff, I finally figured out it was because every reference to the effectful questionTransitions and answerTransitions was deserializing them from disk again. Needless to say, that was not the desired behavior.

After some digging I found System.IO.Memoize, which memoizes expensive IO actions (like deserializing a giant transitions object). Initially this didn't help because I was memoizing too late. So I moved it right to app startup:

startApp :: IO ()
startApp = do
  cachedQt <- eagerlyOnce questionTransitions
  cachedAt <- eagerlyOnce answerTransitions
  run 8080 $ simpleCors $ app cachedQt cachedAt

(Incidentally, most of this stuff is standard servant boilerplate, just tweaked in order to use my cached Transitions.)

The type of eagerlyOnce is

eagerlyOnce :: IO a -> IO (IO a)

Since questionTransitions is IO Transitions, this means that eagerlyOnce questionTransitions is IO (IO Transitions). Since we're in an IO context, the <- means that cachedQt and cachedAt are both IO Transitions (and that they should memoize their values).

(The simpleCors is just middleware that allows our service to handle cross-origin requests.)

Now we can define our Application.

Which again needs the cached transitions as inputs, I am not very happy about the ugly way we're passing them around everywhere, but when I tried to avoid that by e.g. moving all the helpers into the `startApp` function, I got all sorts of cryptic "Couldn't match type" errors, so eventually I gave up and accepted my fate.

It's pretty simple (again, this is all basically servant boilerplate):

app :: IO Transitions -> IO Transitions -> Application
app cachedQt cachedAt = serve api (server cachedQt cachedAt)

And finally we define the server:

server :: IO Transitions -> IO Transitions -> Server API
server cachedQt cachedAt = liftIO $ do
  qt <- cachedQt
  at <- cachedAt
  randomQuestion 4 (randomNextToken qt) (randomNextToken at)

In an IO context it retrieves the cached transitions for the questions and answers, and then it uses them to generate a random Question. It then uses liftIO to lift the Question out of the IO context and into the Server context.

There is a tiny amount of more boilerplate:

api :: Proxy API
api = Proxy

AND THAT'S IT. If you build and run it, you'll end up with a (very fast) service running on localhost:8080:

$ curl http://localhost:8080/question
{"answers":["a rainstorm lasting several times","preventing too many babies the fall leaves","worms from the morning.","conserving water."],"correctAnswer":0,"questionText":"In order of behavior is most important to make life must first"}

The Punchline

After all that work, I spent a couple of hours trying to deploy this to an EC2 machine, failing miserably. The generated executable depends on a bunch of libraries on my system. When I tried to statically include those, the compilation failed. And the EC2 machine was way too underpowered to install stack and build it there. The Internet/StackOverflow was not a lot of help.

At the end of the day, I just rewrote it in flask and deployed that version. :sad_face

(However, it was only because I had a (much faster) flask version that I realized the servant version was way too slow and went down the System.IO.Memoize path, so in that sense it's a good thing!)

The flask version is up and running at http://54.174.99.38/question:

$ curl http://54.174.99.38/question
{"questionText": "Which system?", "answers": ["Absorbing water plants than the air pollution", "It will be healthy", "flood the air pollution", "tying a great gardener."], "correctAnswer": 0}

But it's a cheap EC2 nano instance, so please be gentle.

Next Time

In the third (and final) post, we'll build a quiz webapp that uses this service.

Building a Stupid Data Product, Part 1: The Data (Python)

2016-02-15T07:00:00+01:00

(part 2, part 3)

As I'm not working right now* , I have a surfeit of time to hack on stupid things. In particular, it seemed like a good idea to hack together a stupid data product.

Inspiration (if you can call it such) struck when the Allen Institute released a dataset of elementary school science questions, presumably so that people can train computers to answer them. I am not quite so ambitious, so instead I decided to use them to train a computer to generate new random bogus elementary school science questions.

Using markov chains to generate text this way isn't particularly exciting, so I thought I'd also play around with the other parts of building a data product: creating a backend service that serves up random questions, and creating a single-page app that allows you to take these random quizzes.

(Code, as always, is on GitHub.)

If you know me, you're thinking "backend service, that sounds like a job for Haskell" and also "single-page app, that sounds like a job for PureScript". You're basically right, with a caveat we'll see later, but in this first blog post let's just focus on the data piece.

We'll use a dead simple markov chain model. For each word in the dataset, we make a list of all the words that we see following it. That is, if we had only two questions: "What is love?" and "What the heck?" then our model would say that 'What' can be followed by either 'is' or 'love'. We generate a sentence by picking a starting word and repeatedly choosing a random next word (based on the transitions we learned from the data) until we finish a sentence.

This means our data goal is to generate these transitions. We'll represent them as dicts where the keys are words, and the values are lists of next words:

{ 'What' : ['is', 'the'],
  'is' : ['love'],
  # and so on
}

To start with, we can just download the CSV:

We only care about the 'question' column, so let's just read that in. (As always, use csv.reader, don't try to parse it by hand!)

with open('questions.csv') as f:
    reader = csv.DictReader(f)
    raw_questions = [row['question'] for row in reader]

# raw_questions[0]
# 'Which property of an object is identified using the sense of smell? (A) color (B) odor (C) temperature (D) weight'

Now we're going to actually want two markov chain models. If you look at the data, you'll see that questions and answers look quite different from one another. So we'll generate one set of "question transitions" and another set of "answer transitions".

To that end, we need a way to split a raw_question into a "question" piece and several "answer" pieces. This looks like a job for regular expressions. Using the example above, we might try something like

re.split("\([A-D]\)", raw_question)

which looks for the letters A - D in parentheses, and splits the string where it finds them. Looking at the file, every question has either 3 or 4 answers, which means that the result of that split should have 4 or 5 elements (including the question text).

Let's see where our regex falls down:

split = "\([A-D]\)"

for q in raw_questions:
    if len(re.split(split, q)) not in [4, 5]:
        print(q)
        break

# answer looks like A. 9 yards B. 18 yards C. 50 yards D. 70 yards

So we need a regex for this case, too, how about "\s[A-D]\.\s". If we iterate this process, we end up with a list of regexes that covers all the questions (except one that has no answers):

splits = [
  "\([A-D]\)",    # (A) python (B) haskell (C) javascript (D) ruby
  "\s[A-D]\.\s",  #  A. python  B. haskell  C. javascript  D. ruby
  "\s[1-4]\.\s",  #  1. python  2. haskell  3. javascript  4. ruby
  "\s[A-D]\s",    #  A  python  B  haskell  C  javascript  D  ruby
  "\s[FGHJ]\s",   #  F  python  G  haskell  H  javascript  J  ruby
  "\n [A-D]\s"    #   A python
                  #   B haskell
                  #   C javascript
                  #   D ruby
]

We're almost ready, the other thing we'll do is use a couple of sentinels:

START = "__START__"
STOP = "__STOP__"

We'll add a transition from START to the first word of every sentence, and a transition from the last word of every sentence to STOP. That way we can generate sentences using essentially the following logic:

word = random_word_after(START)
while word != STOP:
  yield word
  word = random_word_after(word)

So, to sum up, our strategy will be:

collect all of the question "sentences" and answer "sentences" separately
use the sentences to generate transition dicts
serialize the transitions, so we can use them in other programs

First, the collecting:

questions = []
answers = []

for q in raw_questions:
    for split in splits:
        pieces = [x.strip() for x in re.split(split, q)]
        if len(pieces) in [4,5]:
            questions.append(pieces[0])
            answers.extend(pieces[1:])
            break   # go on to the next raw_question
    else:           # have you ever seen someone use a for .. else loop?
        print(q)    # me neither!

Now we have a list of questions and a list of answers, so let's turn each into a dict of transitions. We'll use another regex to turn each question (or answer) into words:

re.findall("[^ ?\.,]+|\?|\.|\,", sentence)

That regex looks kind of cryptic, but it's just looking to match either

a period,
a comma,
a question mark, or
a "word" that contains none of the above or spaces

We just need to remember to add the "sentinel" words, and we have our function:

def make_transitions(sentences):
    transitions = defaultdict(list)
    for sentence in sentences:
        # regex looks for "?", ".", "," or groups of characters that aren't
        # any of those, and aren't spaces
        words = [START] + re.findall("[^ ?\.,]+|\?|\.|\,", sentence) + [STOP]
        for prev_word, next_word in zip(words, words[1:]):
            transitions[prev_word].append(next_word)
    return transitions

And then we just need to save them for later use:

q_transitions = make_transitions(questions)
a_transitions = make_transitions(answers)

with open('questions.json', 'w') as f:
    f.write(json.dumps(q_transitions))

with open('answers.json', 'w') as f:
    f.write(json.dumps(a_transitions))

Now we can generate sentences as above:

def next_word(transitions, word):
    return random.choice(transitions.get(word, [STOP]))

def markov_gen(transitions):
    word = next_word(transitions, START)
    while word != STOP:
      yield word
      word = next_word(transitions, word)

For example, to generate a fake question and four fake answers:

In [56]: ' '.join(markov_gen(q_transitions))
Out[56]: 'For which part of the deepest ?'

In [57]: ' '.join(markov_gen(a_transitions))
Out[57]: 'A deer'

In [58]: ' '.join(markov_gen(a_transitions))
Out[58]: 'Old refrigerators contain genes .'

In [59]: ' '.join(markov_gen(a_transitions))
Out[59]: 'producers'

In [60]: ' '.join(markov_gen(a_transitions))
Out[60]: 'Mount St .'

Next Time

In the next post we'll build a web service that generates questions on demand.

* I'm sort of looking for my next thing, if you know of something I'd find interesting, let me know.

Creating a Function Index Using Cycle.js

2016-01-21T09:00:00+01:00

The readers of my book have been clamoring for an index of functions, so that -- for example -- when someone sees me use vector_mean on page 200 they can easily figure out where to find its definition.

It was easy enough (if tedious) to go through the book and create a spreadsheet, but it also seemed an opportunity to build something in Javascript, which I always unreasonably enjoy.

My idea was pretty simple: a text input that you type in, and a table of { function_name, chapter, page } that's filtered (in real-time) by whatever you type. So if you type "add" you get all the functions that contain "add" somewhere in their name (e.g. vector_add).

Simple enough, but also a good opportunity to use/learn some newfangled client-side-virtual-dom-javascript-magic framework.

Given my love of all things Purescript, I built the first version using purescript-halogen. Unfortunately (or fortunately), I don't understand free monads well enough to explain it to you. If you understand Purescript (or Haskell) it is mostly straightforward, save that the halogen library gave me no easy way (or at least no obvious way) to get at the value of a text input on every keyup event, which required a lot of ugly hacking around (and made up the bulk of the time I spent on the project).

Recently I've been poking at cycle.js (whose creator recently wrote an anti-React rant that kind of resonated with me), so I figured I'd try recreating the project using cycle. (It was surprisingly easy and fun, once I got past the self-loathing of using a non-typed language.)

The central idea of cycle is using observables, which are basically lazy streams of values. (If you want types, I believe this is basically the paradigm of Elm, and also there is a purescript-signal library that does this.)

The documentation calls the architecture "Model-View-Intent", but I prefer to think of it as "Intent-Model-View", which is the order in which things happen:

Intent: given some external source (the DOM, AJAX calls), create a lazy stream of events.
Model: given a lazy stream of events, turn it into a lazy stream of states.
View: given a lazy stream of states, turn it into a lazy stream of outputs (in our case, virtual-dom trees)

Each of these can be a pure function (yay!), and the cycle.js machinery hooks these together to create a loop.

In this case, we'll have a text input, and the intent should output its value every time there is a keyup event:

function intent(DOM) {
  return DOM.select('input').events('keyup')
    .map(ev => ev.target.value)
    .startWith('');
}

Hopefully this is pretty straightforward. (The startWith('') just says that our stream should start with an empty string event before any events happen.)

Now the model needs to use that value (our "query") to produce a state. The state should be whatever data we need to produce the output. Here that will be the query itself, as well as the list of functions that satisfy the query:

function model(value$) {
  return value$.map(query => ({
    query,
    entries : indexEntries.filter(entry => (entry.name.indexOf(query) !== -1))
  }));
}

(The $-suffix is a cycle convention (maybe broader) to indicate that a variable is an observable.) This is again pretty simple: given an element of the value$ stream (which is the value of the text input after a keyup), output a state consisting of both the query and the index entries that match it.

Finally, we need the view to turn that state into a virtual-dom representation:

function view(state$) {
  return state$.map(state =>
    div([
      input({type: 'text',
             autofocus: true,
             placeholder: 'Search Query',
             value: state.query}),
      // v-dom table containing the entries
      showEntries(state.entries)
    ]));
}

Given a state, we need a text input with its value set to state.query, and a table displaying the entries. (Yes, this is one of those setups where you use Javascript functions to write HTML, deal with it.) showEntries is not very interesting, but here it is for completeness:

function showEntries(indexEntries) {
  return table([
    thead([
      tr([th('chapter'), th('page'), th('function') ])
    ]),
    tbody(indexEntries.map(entry =>
      tr([td('' + entry.chapter), td('' + entry.page), td(entry.name)])
    ))
  ]);
}

(The table header contains headings, the table body contains one row for each index entry. Apparently td requires a string as its input.)

That's essentially it, all that's left is to hook it all together:

function main({DOM}) {
  return {DOM: view(model(intent(DOM)))};
}

Cycle.run(main, {
  DOM: makeDOMDriver('#app')
});

Here main just takes the DOM source (which is the output of makeDOMDriver) and returns the virtual-dom sink that we've defined. (Or possibly I got "source" and "sink" mixed up here, I don't really understand them.)

Anyway, that's it, that's the whole app (modulo a html file that has an div#app and some hacky CSS and the file that contains the actual "index entries" data).

Because there's lots of ES2015 voodoo, we need to use browserify with all sorts of plugins, and the output is a disgustingly large 600KB Javascript file. That's the price of progress, I guess!

(Code on GitHub.)

Polyglot Twitter Bot, Part 4: PureScript

2015-12-31T09:00:00+01:00

[The fourth in an (at least) 6-part series, all code on GitHub as always.]

Node.js
Node.js + AWS Lambda
Python 2.7 + AWS Lambda
PureScript
PureScript + AWS Lambda
Bonus: PureScript + Twitter Streaming

I know, you're thinking, "I've already read three parts of this series, and I haven't heard one mention of Haskell. Who are you and what have you done with Joel?"

Well, AWS Lambda doesn't support Haskell. (Yet.) Instead we'll work in PureScript, a very Haskell-like language that compiles to Javascript. In particular, we'll be able to (re-)use the Node.js twitter library via PureScript's foreign function interface.

[Big caveat: I am a PureScript newbie. Although the code here works, it's possible (indeed, likely) that it's not well-designed PureScript. And although my explanations reflect my understanding of what's going on, it's possible (indeed, likely) that some of them are totally wrong. Mostly I did this part to help me learn PureScript better. If any PureScript gurus are reading this, I am eager to hear what I could have done better or more idiomatically.]

To start with, install PureScript and its build tool pulp.

Then we need to create a directory and initialize:

$ mkdir purescript-twitter-bot
$ cd purescript-twitter-bot
$ pulp init
$ npm init
$ npm install twitter --save

(It's possible that it's bad form to both pulp init and npm init, but I did both.)

A new PureScript project doesn't include by default a lot of its basic libraries, so we'll need to install the ones we need for this project:

$ pulp dep install --save purescript-console purescript-foreign purescript-arrays purescript-strings  purescript-functions

Now, pulp init should have created a src subdirectory. Go there and create Twitter.purs, where we'll create all of the common types for working with Twitter, as well as the function to initialize the Twitter client.

module Twitter where

import Prelude
import Data.Foreign (Foreign())
import Control.Monad.Eff (Eff())

-- | Effect type for interacting with Twitter.
foreign import data TWITTER :: !

-- | The Twitter client returned by the Javascript `Twitter()` constructor.
foreign import data TwitterClient :: *

What do these things mean? In PureScript functions are (by default) pure. If we want them to have side effects, we need to explicitly declare those side effects. And a function that interacts with Twitter is necessarily impure, since it depends on -- and possibly modifies -- the state of Twitter. So we need to define a TWITTER effect that we can use to mark functions as having Twitter-based side effects (or side inputs). (The bang ! means that TWITTER is an effect.)

In comparison, the * means that TwitterClient is a type, this allows us to define functions that take a TwitterClient as input and functions that return a TwitterClient as output. (The foreign import means that we're not going to define the type within PureScript, but that it's going to refer to something we'll do in Javascript.)

Notably, these are (mostly) just names. We will define a (Javascript-implemented) function that returns a TwitterClient, as well as other functions that take a TwitterClient as input. The type system will simply enforce that any function that wants a TwitterClient gets something that we've identified as a TwitterClient. And that any computation that involves a TWITTER effect can only run in a context that allows TWITTER effects.

(If this all seems new and confusing, that's because it is new and confusing. Bear with me.)

We also need a type to represent Twitter credentials:

type Credentials = {
  consumer_key :: String,
  consumer_secret :: String,
  access_token_key :: String,
  access_token_secret :: String
}

PureScript has easy record types that basically correspond to Javascript objects. That is, the Credentials type is (basically) a Javascript object that has those exact four keys, and whose values are all strings. (However, on the PureScript side it is typed, and you would get an error if you tried to create a Credentials instance with no consumer_key or with a numeric consumer_secret.)

Now we can use the Foreign Function Interface to get an instance of the Twitter client. Let's first handle the PureScript side:

foreign import twitterClient :: forall eff. Credentials -> Eff (twitter :: TWITTER | eff) TwitterClient

Yikes. The foreign import means that we're going to define this function in Javascript. And the type says that this function takes as input a Credentials object and does something that involves a TWITTER side effect and returns a TwitterClient.

(Eff is the monad for specifying native effects, and indicates that this function can only be run in a "context" that allows TWITTER side effects. The forall eff means that the context can allow other side effects as well.)

Now we're ready to write the Javascript side. Create a companion file Twitter.js:

/* global exports */
"use strict";

// module Twitter

var Twitter = require('twitter');

exports.twitterClient = function(credentials) {
  return function() {
    return new Twitter(credentials);
  };
};

That's it. The module Twitter comment is important (I think) and tells the PureScript compiler that this goes with the Twitter module in Twitter.purs. The code simply loads the Node.js twitter library and exports the twitterClient function that we declared in Twitter.purs. Its input is a PureScript Credentials object (which gets translated here into just a plain Javascript object, which is exactly what the Twitter function requires).

The only subtlety is that instead of returning the Twitter client directly, we wrap it in a function of zero arguments. We need to do this whenever our function returns an Eff context. (And conversely, if you create a PureScript function whose return value is an Eff, the generated Javascript function requires an extra call, as we'll see later (and which caused me a lot of confusion)).

Let's also create simple (non-production quality) types to represent Tweets:

type TweetId = String

type Tweet = {
  id :: TweetId,
  user :: String,
  text :: String
}

type Tweets = Array Tweet

So, for us, a tweet has an id, a user, and some text. Obviously the actual data model is a lot richer, but this is all we'll need to build our bot.

Next we need to write the code to interface with the search API. We'll stick this in its own module Twitter.Search which we'll create in src/Twitter/Search.purs.

module Twitter.Search where

import Prelude (Unit())
import Data.Function
import Control.Monad.Eff (Eff())

import Twitter

This looks like the previous set of imports, except that now we also import the Twitter module we just created, as well as the Data.Function module.

You see, PureScript functions are all really functions of a single variable. If you were to have

sum :: Int -> Int -> Int
sum a b = a + b

then sum is really a function that takes an Int and returns a new function Int -> Int. For example sum 2 is the function that adds 2 to any number, and sum 2 3 is really (sum 2) 3.

This means that (naively) we need to write foreign functions the same way:

foreign import sum :: Int -> Int -> Int

with implementation

exports.sum = function(a) {
  return function(b) {
    return a + b;
  }
}

If you have functions with a lot of parameters, this can get ugly really fast. That's where Data.Function comes in, it provides helper functions that allow us to write normal multiple-argument Javascript functions.

foreign import sumImpl :: Fn2 Int Int Int

sum :: Int -> Int -> Int
sum = runFn2 sumImpl

with implementation

exports.sumImpl = function(a, b) {
  return a + b;
}

The declaration Fn2 Int Int Int means sumImpl is a Javascript function of two Int arguments that returns an Int result. And the runFn2 converts it into the usual curried PureScript function. The exposed sum still looks the same as before, so anyone using the function doesn't need to worry about all of these details.

Anyway, back to Twitter searching. The Twitter search API allows you to specify a lot of options, but we'll restrict ourselves to just q [query] and count [number of results]. And we'll provide a helper function that allows callers to just specify the query:

type SearchOptions = {
  q :: String,
  count :: Int
}

searchOptions :: String -> SearchOptions
searchOptions query = {
  q : query,
  count : 15
}

Finally, we're ready to define the search function, using the Data.Function trick from above:

foreign import searchImpl :: forall eff. Fn3
                                         TwitterClient
                                         SearchOptions
                                         (Tweets -> Eff (twitter :: TWITTER | eff) Unit)
                                         (Eff (twitter :: TWITTER | eff) Unit)

search :: forall eff. TwitterClient ->
                      SearchOptions ->
                      (Tweets -> Eff (twitter :: TWITTER | eff) Unit) ->
                      (Eff (twitter :: TWITTER | eff) Unit)
-- the docs suggest not using point-free style here
search client options callback = runFn3 searchImpl client options callback

That is, the search function takes a TwitterClient, some SearchOptions, and a callback (that takes some Tweets as input and does something effectful with them), and does something with a TWITTER effect (and returns no result).

[Incidentally, I would rather not insist that the callback include the TWITTER effect. Our eventual "retweet" callback will, but you could also imagine just a "log to the console" callback that doesn't. However, it caused me problems if I didn't include it, so it's there.]

Then we need to define searchImpl in Search.js, which is not dissimilar to our initial Node.js version:

exports.searchImpl = function(client, searchOptions, callback) {
  // Because `searchImpl` returns a value in the Eff monad, its Javascript implementation
  // needs to return a function of no arguments.
  return function() {
    client.get('search/tweets', searchOptions, function(error, tweets, response){
      var results = tweets.statuses.map(function(tweet) {
        // Map results to our `Tweet` record type.
        // (If `Tweet` wasn't a plain old record type, we'd have to do something
        //  more complicated here.)
        return { id : tweet.id_str, user : tweet.user.screen_name, text : tweet.text };
      });
      // Similarly, because `callback` returns a value in the Eff monad, its
      // Javascript transpilation returns a function of no arguments, which means
      // that to actually *execute* the callback, we need to call the returned function.
      // Not realizing this caused me a lot of grief.
      callback(results)();

      // Because the return type is `Unit`, we just return an empty object.
      return {};
    });
  };
};

We similarly create a Twitter.Retweet module in src/Twitter/Retweet.purs, with just a single function:

foreign import retweetImpl :: forall eff. Fn2
                                          TwitterClient
                                          TweetId
                                          (Eff (console :: CONSOLE, twitter :: TWITTER | eff) Unit)

retweet :: forall eff. TwitterClient ->
                       TweetId ->
                       Eff (console :: CONSOLE, twitter :: TWITTER | eff) Unit
-- the docs suggest not using point-free style here
retweet client tweetId = runFn2 retweetImpl client tweetId

Where retweetImpl is defined in src/Twitter/Retweet.js as

exports.retweetImpl = function(client, tweetId) {
  return function() {
    client.post('statuses/retweet/' + tweetId, function(err, tweet, id) {
      console.log(err || tweet.text);
    });
    return {};
  };
};

Finally, we're ready to do the actual work. (So far we've just been doing the groundwork.) As usual, we create a separate file for credentials, here src/MyCredentials.purs

module MyCredentials where

import Twitter

myCredentials :: Credentials
myCredentials = {
  consumer_key: "...",
  consumer_secret: "...",
  access_token_key: "...",
  access_token_secret: "..."
}

And then we stick the actual work in src/Main.purs. After importing all the stuff we need, we can create a findAndRetweet function.

module Main where

import Prelude
import Control.Monad.Eff
import Control.Monad.Eff.Console
import qualified Data.Array as Array
import Data.Maybe
import qualified Data.String.Regex as Regex

import MyCredentials
import Twitter
import Twitter.Search
import Twitter.Retweet

findAndRetweet :: SearchOptions ->
                  Maybe Regex.Regex ->
                  TwitterClient ->
                  Eff (twitter :: TWITTER, console :: CONSOLE) Unit
findAndRetweet options rgx client = search client options retweetMatches
  where
    -- for each tweet that passes the regex filter, send its id to `retweet`
    retweetMatches tweets = do
      foreachE (filter rgxFilter tweets) (\tweet -> retweet client tweet.id)
    -- if there's a regex, make sure it matches tweet.text
    rgxFilter tweet = case rgx of
      Just pattern -> Regex.test pattern tweet.text
      Nothing -> true

Given a TwitterClient and some SearchOptions, it just runs the search using the callback retweetMatches defined here. And retweetMatches just filters out tweets that don't match the regex (if there is one) and calls retweet for each of the tweets that's left.

Finally, we just need a main function to do the work.

-- This assumes MyCredentials.purs exports myCredentials :: Credentials
main :: Eff (console :: CONSOLE, twitter :: TWITTER) Unit
main = twitterClient myCredentials >>= (findAndRetweet options rgx)
  where
    query = "make \"great again\" -america -filter:retweets"
    options = searchOptions query
    rgx = Just $ Regex.regex "make (.*) great again" flags
    flags = Regex.noFlags { ignoreCase = true }

The type of main indicates that it does something that involves both the CONSOLE side effect and the TWITTER side effect, and that it doesn't return anything. (Unit is similar to other languages' void.)

And >>= is the scary monad bind, which grabs the TwitterClient out of the Eff monad (recall that twitterClient returns an Eff _ TwitterClient) and hands it to the findAndRetweet function, which (after currying the search options and regex) just takes a TwitterClient and does its magic.

If you run this from the command line:

$ pulp run

it should retweet all the things!

Final note: this probably seems like a lot of work. It was a lot of work. But most of the work was creating a (bare-bones, toy) PureScript Twitter library. If you already had such a library (which, in many applications, you would), it would have been a lot less work, and you would just have to have written the code in Main.purs.

Next time we'll get this PureScript version running on AWS Lambda. (Which might take me a couple of days to pull together, happy new year!)

Polyglot Twitter Bot, Part 3: Python 2.7 + AWS Lambda

2015-12-30T12:00:00+01:00

[The third in an (at least) 6-part series, all code on GitHub as always.]

Node.js
Node.js + AWS Lambda
Python 2.7 + AWS Lambda
Purescript
Purescript + AWS Lambda
Bonus: Purescript + Twitter Streaming

Lambda also allows Python functions, although only Python 2.7. In practice, this won't affect us much, as we'll just

from __future__ import print_function

and cross our fingers that our bot will do the right thing where Unicode is concerned. (Or, at least, won't crash with a UnicodeEncodeError.)

Now, because Python isn't oriented around asynchronous callbacks, we don't need to worry about context.succeed and context.fail, instead we can just write the usual Python function that's done when it's done.

To work with Twitter, we'll use the Twython library. As before, we'll need to "deploy" a zip file that contains all of the dependencies we need, which means we need to install Twython into our project directory.

Create a directory for your Python bot and in that directory run

$ pip install twython -t .

which should install all the Twython files right there. (There are various subtleties, for example, if you are using virtualenv, or if your Python is installed using Homebrew, which you should read about here, but which you should not expect me to understand or explain.)

Now we're ready to write our bot. In index.py (because why not?), let's start with the imports we need:

from __future__ import print_function
from twython import Twython
from twython.exceptions import TwythonError
import re
import json

As always, we need credentials, but we don't want them checked into our code. One way is to create a credentials.json file

{
  "consumer_key": "...",
  "consumer_secret": "...",
  "access_token_key": "...",
  "access_token_secret": "..."
}

which our script can just load:

with open('credentials.json') as f:
  credentials = json.loads(f.read())

After which we can initialize the client:

client = Twython(credentials["consumer_key"],
                 credentials["consumer_secret"],
                 credentials["access_token_key"],
                 credentials["access_token_secret"])

Of course, we also need to create our query and regular expression:

query = 'make "great again" -america -filter:retweets'
rgx = r"make (.*) great again"

And then we just have to create the handler:

def handler(event, context):
    results = client.search(q=query)
    for tweet in results["statuses"]:
        text = tweet["text"]
        # re.search matches anywhere in the string; re.I means case-insensitive
        if re.search(rgx, text, re.I):
            print(tweet["text"])
            # client.retweet will raise an error if we try to retweet a tweet
            # that we've already retweeted. to avoid having to keep track, we
            # just use a try/except block
            try:
                client.retweet(id=tweet["id"])
            except TwythonError as e:
                print(e)

And that's the bot! We just need to zip it up:

$ zip -r twitter.zip *

and go through the exact same "create a lambda function" process as before. (Make sure to choose "Python 2.7" as the runtime.) Probably (certainly) you don't need both the Node bot and the Python bot active at the same time, so make sure to disable the event source for one of them.

Next post we'll dive into the crazy world of Purescript!

Polyglot Twitter Bot, Part 2: Node.js + AWS Lambda

2015-12-29T19:00:00+01:00

[The second in an (at least) 6-part series, all code on GitHub as always.]

Node.js
Node.js + AWS Lambda
Python 2.7 + AWS Lambda
Purescript
Purescript + AWS Lambda
Bonus: Purescript + Twitter Streaming

AWS has a recent-ish Lambda product, which lets you upload functions and then have them run on demand (or on a schedule), without having to actually run a server. There's a lot of interesting things you can do with this functionality, one of which is "run a Twitter bot".

These days Lambda allows Node.js, Python, or Java functions, but for now we'll modify and use the Node.js bot we built previously.

The only trick is that we need to fit it into the Lambda execution model.

Lambda expects our Node module to export a handler function with the following signature:

exports.handler = function(event, context) {
   // do something
};

Here event contains the parameters for the function invocation, and context contains Lambda-specific details. Our bot function won't take any parameters, since it has everything it needs (query, credentials, etc...) hardcoded in it. The context object contains succeed and fail methods that tell Lambda that the function is done running. (This is why we included them last time.)

So a handler that would work here is just

exports.handler = function(event, context) {
  searchAndTweet(context.succeed, context.fail);
};

Now, if your function doesn't use any fancy libraries, you can simply type the code into the AWS console. However, here we're using the twitter library, so we can't. Instead we have to create a deployment package, which is just a zip file.

$ ls
credentials.js  index.js  node_modules  package.json

In particular, we need to include the node_modules directory, which contains the Twitter library and its dependencies.

$ zip -r twitter.zip *

This gets me a ~1.3MB zip file. Now go to your AWS console and click "Create a Lambda Function".

"Skip" the "select blueprint" step, give your function a name, and upload the zip file.

The default handler name index.handler should work (index means to look in index.js, .handler means to use the function called that), and then create or choose a "basic execution" role.

The 128MB should be plenty of memory, but you might want to up the timeout to 10 seconds or so.

Finally, click "Create Function". At this point you want to "Save and Test" to make sure it works.

Finally, you probably want it to run on a schedule, so choose the "Event Sources" tab and click "Add event source". Go to "scheduled event" and enable it to run every 5 minutes (or every 15 if you want).

(I am not much of an AWS pricing guru, but my best estimate is that this Lambda function should cost you approximately $0.)

And that's it, your bot is done! Next time we'll do it in Python.

Polyglot Twitter Bot, Part 1: Node.js

2015-12-29T12:00:00+01:00

[The first in an (at least) 6-part series, all code on GitHub as always.]

Node.js
Node.js + AWS Lambda
Python 2.7 + AWS Lambda
Purescript
Purescript + AWS Lambda
Bonus: Purescript + Twitter Streaming

Like most of you, I've long dreamed of making a Twitter bot. And also like most of you, I've been doing a lot of Node.js recently. So I thought I'd take the first stab at writing my Twitter bot in Node. (Also, this will lay the groundwork for doing it in Purescript later.)

In particular, I wanted to create the make_greatagain bot, which would look for tweets containing "MAKE ___ GREAT AGAIN" constructions and retweet them. (But which skips tweets containing "MAKE AMERICA GREAT AGAIN", I'm looking for riffs on the original, not the original itself.)

Tweets by @make_greatagain

To start with, you should probably have Node installed. (I'll wait.) Then create a directory, and initialize a new project:

mkdir twitter-bot-node
cd twitter-bot-node
npm init

Just accept all the default options for npm init, I don't know what they mean either.

Now, if we're going to talk to Twitter, we should install the Node Twitter module.

npm install twitter --save

At this point you should create a Twitter account for your bot and get its credentials. After creating the account and logging in, go to apps.twitter.com and click on "Create New App". Give it a name and a description, and accept the terms of service. Then go to the "Keys and Access Tokens" tab and click "Create My Access Token". You should now have a consumer key, a consumer secret, an access token, and an access token secret. We need those, but KEEP THEM SECRET.

Now, we're ready to create our index.js. We start by loading the Twitter library and initializing it with our credentials:

var Twitter = require('twitter');
var client = new Twitter({
  consumer_key: "...",
  consumer_secret: "...",
  access_token_key: "...",
  access_token_secret: "..."
});

NOTE: if you are committing this code to GitHub, DO NOT CHECK IN THE CREDENTIALS. One approach is to stick them in credentials.js, like

module.exports = {
  consumer_key: "...",
  consumer_secret: "...",
  access_token_key: "...",
  access_token_secret: "..."
};

and then in index.js just do

var credentials = require('./credentials');
var client = new Twitter(credentials);

and then make sure to add credentials.js to your .gitignore.

Now, we want to find Tweets of the given form. For my example, that's

var query = 'make "great again" -america -filter:retweets';
var rgx = /make .* great again/i;

(Hopefully, your Twitter bot will do something different.)

The query is the actual query we'll send to Twitter. It looks for Tweets that contain both "make" and "great again" but not "america". And it ignores retweets. Since that search could (in theory) return irrelevant tweets (e.g. "great again doesn't make sense"), there's also a regex that we'll use as a client-side check.

Now, the Node model is asynchronous, which means we need to program with callbacks. That is, to search, we need to do something like

client.get('search/tweets', {q: "node.js"}, function(err, tweets, response) {
  if (err || !tweets.statuses) {
    console.log(err);
  } else {
    tweets.statuses.forEach(function(tweet) {
      console.log(tweet.user.screen_name + " " + tweet.text);
    });
  }
});

This code will kick off a search for "node.js" and then immediately go on to whatever code comes next. Meanwhile, whenever the search returns, the provided callback will be called, either logging the error or printing out the returned tweets.

Now in our code we want the callback to retweet each of the returned tweets. However, if we try to retweet a tweet we've already retweeted, we'll get an error. This means we either need to keep track of all the tweets we've already retweeted or else handle those errors intelligently. The second is a lot easier.

In order to retweet, we just need to post the tweet id to the retweets endpoint. If you inspect the returned tweets, they have both an id field (which is a number) and an id_str field (which is a string). For precision-related reasons (I assume), Javascript mangles the numeric ids, so we'll need to use the string version.

All of which results in a function that looks like

// Runs a Twitter search for the specified `query` and retweets all the results.
function searchAndTweet(succeed, fail) {
  console.log("search and tweet");
  client.get('search/tweets', {q: query, count: 15}, function(err, tweets, response) {
    if (!tweets.statuses) {
      fail(err);
    }

    tweets.statuses.forEach(function(tweet) {
      // Make sure we match the regex.
      var match = tweet.text.match(rgx);
      if (match) {
        var tweetId = tweet.id_str;
        client.post('statuses/retweet/' + tweetId, function(err, tweet, id) {
          // Will return an error if we try to retweet a tweet that we've already
          // retweeted.
          console.log(err || tweet.text);
        });
      } else {
        // consider doing something for no match
      }
    });
    succeed("success");
  });
}

Why do we pass in the succeed and fail callbacks? That's a story for the next post. (Spoiler: it involves AWS Lambda.) In the meantime, you can just pass in console.log for both.

Now, all that's left is to run your Twitter bot. We can use setInterval to make it run every 5 minutes:

setInterval(function() {
  searchAndTweet(console.log, console.log);
}, 5 * 60 * 1000);

And then if you simply

$ node index.js

your bot will start running. Of course, you probably don't want to keep it running locally on your computer all the time. We'll deal with that in the next post.

Stupid Itertools Tricks for Data Science

2015-08-23T12:00:00+02:00

(This is a blog post version of my PyData Seattle talk, slides and code at the link.)

In my last post

Haskell-Style Fibonacci in Python

2015-07-07T12:00:00+02:00

If you've ever done a tech interview, you're probably familiar with the Fibonacci sequence:

1, 1, 2, 3, 5, 8, 13, ....

where each number is the sum of the previous two. A relatively simple (and relatively overused) interview problem is to write a function that returns the n-th Fibonacci number.

Recursive Python

The most intuitive implementation is recursive:

def fib(n):
  """the profoundly inefficient recursive implementation"""
  if n in [0, 1]:
    return 1
  else:
    return fib(n - 1) + fib(n - 2)

It works in a pretty obvious fashion.

>>> map(fib, range(10))
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

However, it's shockingly inefficient. To compute fib(10) we need to compute fib(9) and fib(8). To compute fib(9) we need to compute fib(8) (again) and fib(7). And so on.

The moral of the story is that to compute fib(10) we end up making 177 calls to fib. To compute fib(20) we end up making 21,891 calls to fib. And to compute fib(30) we end up making 2.7 million calls to fib, which ends up taking almost a full second on my laptop.

One solution is memoization, in which we remember previously computed values. We won't get into that here.

Iterative Python

Another solution is an iterative approach:

def fib(n):
  """the iterative implementation"""
  x, y = 0, 1
  for _ in range(n):
    x, y = y, x + y
  return y

This is much more efficient -- to compute fib(n) we just do O(n) operations. At the same time, it's much less clear what it's actually computing. If I didn't tell you it was fib you might have to stare at it for a while to figure out exactly what it was doing. And even knowing that it's fib it's probably not obvious that it's implemented correctly.

Haskell

Haskell, in case you don't know, is everyone's favorite pure functional programming language. In particular, it embraces laziness in which values are computed only as needed.

This means we can compute the (infinite) sequence of Fibonacci numbers as

fibs :: [Int]
fibs = 1 : 1 : zipWith (+) fibs (tail fibs)

Prelude> take 10 fibs
[1,1,2,3,5,8,13,21,34,55]

You should understand this definition as "the sequence that starts with 1, followed by 1, followed by the sequence of numbers obtained by "zipping" together the sequences fibs and tail fibs (that is, all of the elements of fibs after the first) and adding together each corresponding pair of elements.

Which means the third element of the sequence is the first element (of fibs) plus the second element (which is the first element of tail fibs). The fourth element is the second element (of fibs) plus the third element (which is the second element of tail fibs). And so on. This is precisely the definition of Fibonacci.

This definition may seem circular, since we're using fibs to define fibs.

After all, the following sort of thing leads to infinite recursion:

def fibs(x=0, y=1):
  return [y] + fibs(y, x+y)

>>> fibs()[0]
....
RuntimeError: maximum recursion depth exceeded

But in the Haskell version, because of laziness, the elements of fibs will only be evaluated as needed. To compute the third element, we only need to know the first two elements, which we already do by the time we're trying to compute the third element, and so on.

Appreciate its stark, mathematical beauty! (Learn Haskell!)

Laziness in Python

We can achieve laziness in Python using generators. One way is list comprehensions in parentheses.

>>> num = (i for i in range(3))
>>> num
<generator object <genexpr> at 0x7f5efc33bd70>
>>> num.next()
0
>>> num.next()
1
>>> num.next()
2
>>> num.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Each element of num is computed only on demand.

We can also use functions with yield:

>>> def to3():
...   yield 1
...   yield 2
...   yield 3
...
>>> [x for x in to3()]
[1, 2, 3]

This is how we'll implement the Haskell-style Fibonacci.

itertools

The Haskell implementation used tail (to get the elements after the first) and take (to get a certain number of elements from the front). Python doesn't have those, so we'll need to implement our own versions.

The itertools module contains some helpers for working with laziness. We'll need islice which allows us to slice a new generator out of an old one.

from itertools import islice

def tail(iterable):
  """return elements from 1 to forever"""
  return islice(iterable, 1, None)

def take(n, iterable):
  """return elements from 0 to n in a list"""
  return list(islice(iterable, 0, n))

For our Python 2.7 version we'll also need imap, which is simply the lazy version of map.

As a warmup, let's create an infinite sequence using the iterative approach:

def fibs():
  x, y = (0, 1)
  while True:
    yield y
    (x, y) = (y, x + y)

Which we can use like:

>>> take(10, fibs())
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Haskell-style in Python

Recall the Haskell implementation:

fibs = 1 : 1 : zipWith (+) fibs (tail fibs)

It's not hard to create the Python 2.7 equivalent

from itertools import imap   # lazy map
from operator import add     # add(x, y) = x + y

def fibs():
  yield 1
  yield 1
  for n in imap(add, fibs(), tail(fibs())):
    yield n

The logic is exactly the same. The first two elements both equal 1. After that we lazily add together the corresponding elements of fibs() and of tail(fibs()):

>>> take(10, fibs())
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

In Python 3.3+, this gets even simpler. First map is already lazy, so there's no reason to use imap. And second, 3.3 introduces yield from which allows us to replace the clunky

for x in xs:
  yield x

with

yield from xs

This means that the Python 3.3+ implementation is simply:

def fibs():
  yield 1
  yield 1
  yield from map(add, fibs(), tail(fibs()))

Which we could one-line as

def fibs(): yield 1; yield 1; yield from map(add, fibs(), tail(fibs()))

Warning!

Despite appearing Haskell-like, this version is basically as inefficient as the original recursive version. That's because every time fibs() is called recursively, Python redoes from scratch all the work to generate the sequence.

Whereas in Haskell things are immutable, which means that there's only a single fibs hanging around. Once its first few elements have been computed, they never have to be computed again.

So while our Python version is clever, it's also impractical.

A Final Comparison

Compare again

def fibs(): yield 1; yield 1; yield from map(add, fibs(), tail(fibs()))

with the Haskell

fibs = 1 : 1 : zipWith (+) fibs (tail fibs)

It's practically the same code! (Yes, the one-line version is hideous Python, I just did that so it would be more comparable with the Haskell version.)

I showed it to one of my friends, who was so impressed that he said "I have no idea what that Python code does".

Next time you get asked Fibonacci as an interview question, consider using this version, and let me know what happens!

Update!!!!!!

As was pointed out in the comments, one can use itertools.tee to split a generator into multiple (efficient) copies. This means that the following slight modification:

def fibs():
  print("a new fibs")
  yield 1
  yield 1
  fibs1, fibs2 = tee(fibs())
  yield from map(add, fibs1, tail(fibs2))

is incredibly efficient.

On The Mathematics of Spot It!

2015-06-12T12:00:00+02:00

Last weekend we went to a party where one of the other attendees brought Spot It! Frozen for her kids. It's a simple game with circular cards, each of which has 8 pictures in it, most of them Frozen-themed.

The setup is that any two cards in the deck have exactly one picture in common. There are various sets of rules, but they all boil down to some variation of "deal two cards, try to spot the common element."

To my surprise / delight, Madeline picked up the game very quickly and was quite good at it, despite her not knowing any of the Frozen characters on account of having never seen the movie. [That's right, I am Dad of the Year, hold your applause until the end.]

So I bought her her own set (original edition) and we started playing it at home. The more I played it, the more I puzzled over its mathematics. How would one go about designing such a deck?

I was unable to figure it out off the top of my head, so I Googled it. Sure enough, I was not the first one to ask this question, although most of the answers I found were not terrifically understandable. I finally achieved enlightenment through this StackExchange answer although it wasn't as clear as I liked. Also, it didn't provide the code to generate the decks, which I decided was a worthwhile exercise. (As always, the code is on GitHub.)

The mathematics involves finite projective planes, which I learned about so that I could explain them to you.

Projective Planes

In run-of-the-mill planes (the kind you likely studied in geometry class), every two distinct points determine exactly one line. And every two distinct lines intersect in either zero points (if they are parallel) or one point (if they are not parallel).

A projective plane contains extra points at infinity where parallel lines can intersect. This means that in a projective plane, any two distinct lines intersect in exactly one point.

One way to make this happen is to -- for each slope -- add a point at infinity where all the lines of that slope intersect. That is, add an "infinity 0" where all horizontal lines intersect, an "infinity 1" where all lines with slope 1 intersect, an "infinity infinity" where all the vertical lines intersect, and so on. Then, so that any two distinct points have exactly one line between them, add a new "line at infinity" that goes through all the infinities.

Finite Projective Planes

Although you are probably not used to thinking about finite "planes", we can do something similar for them.

Choose some prime number n, and consider the n x n grid of points:

def ordinary_points(n):
    return [(x, y)
            for x in range(n)
            for y in range(n)]

(We'll explain why we chose n to be prime in a bit.)

In this finite plane we do arithmetic mod n, so that (for example) the set of points with y = 0 is in fact a horizontal line that "wraps around" from (n - 1, 0) to (0, 0).

It turns out that there are n + 1 ordinary lines through (0, 0):

slope 0: goes through (1, 0), (2, 0), ...
slope 1: goes through (1, 1), (2, 2), ...
slope 2: goes through (1, 2), (2, 4), ...
slope 3: goes through (1, 3), (2, 6), ...
...
infinite slope: goes through (0, 1), (0, 2), ...

and every point that's not (0, 0) lies on exactly one of these lines.

This is one place where the prime-ness of n matters. For instance, if we'd chosen n = 4, then the line with slope 0 [(0, 0), (1, 0), (2, 0), (3, 0)] and the line with slope 2 [(0, 0), (1, 2), (2, 0), (3, 2)] both pass through (0, 0) and (2, 0), but clearly they're not the same line. n being prime ensures the "two points lie on exactly one line" condition.

Each ordinary (non-vertical) line is defined by its slope and its intercept. For example, there is a line through (0, 0) with slope 1, which passes through (0, 0), (1, 1), (2, 2), and (3, 3). And there is a line through (0, 1) with slope 1, it passes through (0, 1), (1, 2), (2, 3), and (3, 0). [Remember that we're doing arithmetic mod n.] Each vertical line is defined just by its x-coordinate.

Again we have the problem that parallel lines don't intersect, and again we'll solve it by adding "points at infinity", one for each slope. We'll represent the "infinity" with slope 1 just as the number 1, and we'll represent the "vertical infinity" as the unicode u"∞".

def points_at_infinity(n):
    """infinite points are just the numbers 0 to n - 1
    (corresponding to the infinity where lines with that slope meet)
    and infinity infinity (where vertical lines meet)"""
    return range(n) + [u"∞"]

Now we just need to make sure the correct infinity belongs to each line:

def ordinary_line(m, b, n):
    """returns the ordinary line through (0, b) with slope m
    in the finite projective plan of degree n
    includes 'infinity m'"""
    return [(x, (m * x + b) % n) for x in range(n)] + [m]

def vertical_line(x, n):
    """returns the vertical line with the specified x-coordinate
    in the finite projective plane of degree n
    includes 'infinity infinity'"""
    return [(x, y) for y in range(n)] + [u"∞"]

We also need to add another line that goes through the points at infinity:

def line_at_infinity(n):
    """the line at infinity just contains the points at infinity"""
    return points_at_infinity(n)

Are You Sure About The "Two Lines One Point"?

I am. But let's prove it. Imagine we have two different lines. We want to prove that they intersect in exactly one point. We have several types of lines, so we'll need to consider every possible combination of cases:

two ordinary lines with the same slope:

Say we have the lines (m1, b1) and (m1, b2). They intersect at an ordinary point if there is some x so that m1 * x + b1 = m1 * x + b2; that is, if b1 = b2. But since they're different lines, necessarily b1 doesn't equal b2. So these lines just intersect at "infinity m1".

two ordinary lines with different slope:

Say we have the lines (m1, b1) and (m2, b2). They clearly don't intersect at infinity, since one passes through "infinity m1" and the other through "infinity m2". They intersect at an ordinary point precisely if

m1 * x + b1 = m2 * x + b2 (mod n)

or if

(m1 - m2) * x = b2 - b1 (mod n)

Because n is prime (this is another place where the prime-ness is important), it turns out there is a unique x for which this is true, so that's where they intersect:

(x, (m1 * x + b1) mod n) = (x, (m2 * x + b2) mod n)

ordinary line (m1, b1), vertical line through x.

It's easy to see they intersect exactly at (x, m1 * x + b1)

ordinary line (m1, b1), line at infinity.

Again, it's easy to see that they intersect exactly at "infinity m1".

vertical line through x1, vertical line through x2.

Intersect exactly at "infinity infinity"

vertical line through x, line at infinity.

Intersect exactly at "infinity infinity"

What the Hell?

This has a point, I promise. Let's pick a n, say 7. The corresponding projective plane has 57 points (7x's * 7y's + another 8 at infinity) and 57 lines (8 slopes * 7 intercepts + line at infinity). Each line passes through 8 of the points, and any two lines intersect in exactly one point. Now mentally translate:

point <-> picture

line <-> card

passes through <-> contains

projective plane <-> deck

intersect in <-> have in common

Translated, the deck has 57 cards and 57 (distinct) pictures. Each card contains 8 pictures, and any two cards have in common exactly one picture. That's exactly the game. (Actually, for reasons unknown to the Internet, the version of the game you buy only has 55 cards. But our version has 57 cards.)

How do we create a deck? First, let's create a couple of functions to collect all of the points and lines:

def all_points(n):
    return ordinary_points(n) + points_at_infinity(n)

def all_lines(n):
    return ([ordinary_line(m, b, n) for m in range(n) for b in range(n)] +
            [vertical_line(x, n) for x in range(n)] +
            [line_at_infinity(n)])

This works, the only problem is the cards are not all that nice looking:

In [x]: random.choice(all_lines(7))
Out[x]: [(0, 4), (1, 5), (2, 6), (3, 0), (4, 1), (5, 2), (6, 3), 1]

In [y]: random.choice(all_lines(7))
Out[y]: [(0, 5), (1, 5), (2, 5), (3, 5), (4, 5), (5, 5), (6, 5), 0]

Indeed, they have the "picture" (1, 5) in common. We should probably give the pictures nicer names. The following function will use the names you pass in:

def make_deck(n, pics):
    points = all_points(n)

    # create a mapping from point to pic
    mapping = { point : pic
                for point, pic in zip(points, pics) }

    # and return the remapped cards
    return [map(mapping.get, line) for line in all_lines(n)]

And now it's easy to play your own game of Spot It! in the shell:

def play_game(deck):
    # make a copy so as not to muck with the original deck, then shuffle it
    deck = deck[:]
    random.shuffle(deck)

    # keep playing until fewer than 2 cards are left
    while len(deck) >= 2:
        card1 = deck.pop()
        card2 = deck.pop()
        random.shuffle(card1)  # shuffle the cards, too, to simulate that
        random.shuffle(card2)  # they might face different directions

        # find the matching element
        match, = [pic for pic in card1 if pic in card2]

        print card1
        print card2

        guess = raw_input("Match? ")
        if guess == match:
            print "correct!"
        else:
            print "incorrect!"

For example, if I give it animal names, a play of the game looks like:

In [x]: play_game(deck)
['Crab', 'Baboon', 'Rat', 'Jackal', 'Bear', 'Buffalo', 'Llama', 'Sheep']
['Snake', 'Yak', 'Jackal', 'Otter', 'Kouprey', 'Wolf', 'Rook', 'Ox']
Match? Jackal
correct!
['Narwhal', 'Coyote', 'Dog', 'Wolf', 'Dotterel', 'Rat', 'Mosquito', 'Tarsier']
['Gnu', 'Dogfish', 'Sheep', 'Porcupine', 'Mosquito', 'Heron', 'Tapir', 'Rook']
Match? Narwhal
incorrect!

and so on. And now you know how the game works. Maybe next time we'll do it in Haskell!

Data Science From Scratch: First Principles with Python

2015-04-26T12:00:00+02:00

I am super-excited to announce that the book I've been working on for more than the last year, Data Science from Scratch: First Principles with Python is finally available! (buy from O'Reilly, use discount code AUTHD to save some money) (buy from Amazon).

My experience learning and teaching data science was that there were two primary paths:

The Math Path: "So you want to be a data scientist? Sure, the first thing you need to know is matrix decompositions. How well do you remember your measure theory?"
The Tools Path: "So you want to be a data scientist? Great, here's the most important libraries to know. How well do you know R?"

Although I am myself a "math person", the first approach never resonated with me. The fun of data science for me has always been working with data. At the same time, I've never been thrilled with the second approach -- it's a good way to start doing data science without ever really understanding what you're doing.

My ideal would be a "third way" between these approaches:

understanding the behavior of the most common tools by working through a solid-but-less-than-textbook-rigorous understanding of the math behind them, and
implementing simplified versions of them from scratch to understand exactly what it is they're doing.

Hence Data Science from Scratch. It's got math, but only as much as is totally necessary. It's got scraping and cleaning and munging. It's got machine learning. It's got databases and MapReduce. Necessarily it doesn't go deep into any of these, but I like to think it establishes a broad, solid foundation for someone who knows some math and some programming but is not (necessarily) an expert at either.

Many technical books (I won't name names) explain things in their text and then dump pages of hard-to-follow code at you that you are expected to puzzle through. I spent a lot of time trying to write clean code that illuminated the concepts on its own and that reinforced the ideas from the text. As is the current fashion these days, all of the code and data is on GitHub, if you'd like to get a sense of what the book is about.

If you are interested in the topic, I encourage you to check it out, write a review, and let me know what you think! (You can see the full table of contents on the O'Reilly page.)

2014 Year in Review

2015-01-15T19:09:00+01:00

2014 was a crazy year, mostly because I took on two very large projects either of which would have made for a pretty crazy year.

1. The Book

I am writing a book. (Yes, another one.) Actually, the first draft is done, and I am right now revising it based on the final set of technical reviews. This one has a real publisher and everything, which means (I hope) that for a change I won't have to sell it out of the trunk of my car in the parking lots at gun shows.

The book will (likely) be called Data Science from Scratch: First Principles with Python, which is a pretty good description of what the book is about. It covers everything from statistics to probability to linear algebra to regression to machine learning, all from first principles. That means no numpy, no scipy, no scikit-learn, no nothing. Every time we need an algorithm, we implement the algorithm.

As a recovering mathematician¹, I have a natural inclination toward "from scratch" approaches², which was only complicated by the fact that I didn't actually know how to do most of the things in the book "from scratch", which meant that I had to learn them well enough not only to use them but to teach them.

There is an old joke (or possibly I made it up and it’s a new joke, I'm not sure):

"How I hate writing!"
"I thought you were a writer."
"Indeed, I think you have to be a writer to hate writing as much as I do."

I am here to tell you that this joke is completely true. (I'm also here to tell you that as soon as this book is done I'll probably start on the next one.)

As the book gets closer to publication I will have plenty more to say about it. To that end I've "rebooted" my website / blog and hope to use it more frequently going forward.

2. Google

In early 2014 I got a call from a Google recruiter who wanted to know whether I'd like to interview for an engineering job at Google. Despite the fact that I was in no shape to pass a Google interview, I eventually agreed and spent two months worth of free time practicing whiteboard coding and teaching myself computer science. (Two months worth of free time that I should really have spent working on my book, or possibly paying attention to my family.)

Interestingly, writing a [good technical] book requires you to put a lot of thought into how to most clearly communicate ideas through code, which also makes for a valuable technical interview skill, and I think I probably interviewed much better on account of the writing I'd done at that point.

I probably have a lot more to say about the interview process at some point in the future, but the moral of the story is that I got the job and started at Google in late August, at which point I had to frantically learn both C++³ and the Google way of writing software, all the while struggling to finish the book.

My hope is that 2015 will be slightly less crazy, but probably it won't be.

1. You never actually recover from mathematics.

2. I once took a math course where on the first day the professor stated how pleased he was that we'd proven a certain theorem the previous semester, since that meant we would be able to [in a "moral" sense] use it in his class.

3. My resume did have C++ listed on it, but under the "programming languages that I have heard of" section.

T-Shirts, Feminism, Parenting, and Data Science, Part 2: Eigenshirts

2013-06-24T20:37:00+02:00

(You might want to read Part 1 first.)

When last we left off, we'd built a model using shirt colors to predict boy-ness / girl-ness.

Our second attempt will involve the shirt images themselves (sort of). For our purposes, computer images are made up of pixels, each of whose color is determined by specifying red, green, and blue values between 0 and 255. So if you have an image with N pixels, you can think of it as a point in 3N-dimensional space, all of whose coordinates lie between 0 and 255.

And as before, we can build a linear model to classify points in space using logistic regression. The trick here is that the images have different sizes (and hence different numbers of pixels). So as a first step, we'll rescale every image to 138 pixels x 138 pixels = 19,044 pixels. (A lot of our images are this size, and the rest are mostly larger, which is why I chose it.) This will give us a representation of each t-shirt image as a point in 57,132-dimensional space. (Visualizing 57,132-dimensional space is tricky, so don't feel bad if you can't do it.)

Our dataset only contains about 1,000 shirts, which means that a 57,000-dimensional classifier would learn to identify every shirt in the test dataset rather than figure out what distinguishes the boys shirts from the girls shirts. This means we need to do some sort of dimensionality reduction to get our t-shirt images into a much lower-dimensional space.

Here we'll use Principal Component Analysis, which finds the direction (in 57,132-dimensional space) that accounts for the largest amount of variance in the dataset. It then subtracts out this direction, finds the most-variant-direction of the new dataset, and so on, until it has enough components.

(As always, code is on GitHub.)

I ended up using 10 components, which gives a representation of each t-shirt as just 10 numbers, representing the projection of the (57,132-dimensional representation of the) shirt onto the first 10 principal components, each of which is itself a vector in 57,132-dimensional space. For instance, the first principal component is the 57,132-element vector

[0.0002334, 0.00029256, 0.00042805, ... , 0.00051605]

By thinking of this as a vector of 19,044 rgb triplets, and by rescaling it so that its smallest component is 0 and its largest component 255, we can convert it into an image of an eigenshirt representing the "essence" of this component. Shirts with a large value for the first component will tend to be "similar" to this eigenshirt. Shirts with a large negative value for the first component will tend to be "similar" to its color-inverted "anti-eigenshirt". [We could have just as easily picked the "anti-eigenshirt" as the eigenshirt and flipped the signs of the components.]

The below table shows, for each of the 10 principal components, the eigenshirt, the shirt with the largest component value, the shirt with the closest-to-zero value, the shirt with the largest negative component value, and the "anti-eigenshirt".

https://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.com https://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.com https://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.com https://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.com https://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.comhttps://joelgrus.com

Eigenshirt	Most Eigenshirty	Not Eigenshirty	Most Anti-Eigenshirty	Anti-Eigenshirt

If I were to try to give qualitative descriptions of these ten components, I guess they would be something like:

Component 0: White -\> Black
Component 1: Orange -\> Blue
Component 2: Dark sleeved / white sleeveless -\> White sleeved / dark sleeveless
Component 3: Wide dark / narrow white -\> Narrow dark / wide white
Component 4: ?
Component 5: Green -\> Purple
Component 6: White trim / dark shirt -\> Dark trim / white shirt
Component 7: https://joelgrus.comwhite sleeveless -\> White long sleeve / dark sleeveless
Component 8: White shirt / dark print -\> Dark shirt / white print
Component 9: ?

The Principal Component representation of each shirt is a 10-dimensional vector representing (roughly) where it fits on each of these spectra. For instance, the monkey shirt

is represented by the vectorhttps://joelgrus.com

[ -9313, 10067, -149, -4013, -2147, 1574, -296, -954, 1729, -196]

the biggest components of which are "orange" (eigenshirt #1), "dark" (anti-eigenshirt 0), and "narrow" (anti-eigenshirt 3).

If we try to reconstruct the image using just these ten components, we get

which seems to have captured orange, short sleeve, and dark graphic. You certainly can't tell it's a monkey, though.

Predicting

If we try to predict "boy shirt or girl shirt" using just these 10 components, we get a model that's 93% accurate on the test set. The coefficients (multiplied by 10,000, since they're small) look like:

Component 0: -2.71 (eigenshirt is girlish)\ Component 1: -2.56 (girlish)\ Component 2: 3.55 (boyish)\ Component 3: 0.53 (weakly boyish)\ Component 4: -0.56 (https://joelgrus.com Component 5: 5.43 (boyish)\ Component 6: -15.9 (very girlish)\ Component 7: -4.68 (girlish)\ Component 8: 2.73 (boyish)\ Component 9: -2.14 (girlish)

As before, we can look at how the shirts are distributed as a function of the score they get from the model:

![shirts_by_scorhttps://joelgrus.comcom/wp-content/uploads/2013/06/shirts_by_score1.png)

The miscategorized shirts generally have low (close to 0) scores, except for one particularly "girly" boys shirt that we'll see below. https://joelgrus.com Superlatives

*Girliest Girlhttps://joelgrus.comsed on shape and colors)

Girliest Boy https://joelgrus.comgain)

Boyiest Boy (da Bears)

Boyiest Girl (same one as last time!)

This is all very interesting and hints at Platonic ideal shirts (the philosophical details of which are out of scope for this blog). And clearly it does a much better job of predicting "boy shirt or girl shirt" than our previous color-based attempt. But whereas everyone knows about colors (except for the color-blind, of course), most people are unfamiliar with "eigenshirts" and will accuse you of having made them up just in order to have something to blog about. In particular, the girl who works at Gap Kids was entirely unimpressed with this model, and said that I needed to either buy something or leave the store.

Were I really committed to this model, I'd probably do more work to get the images comparable to each other so that not only were they the same size but the shirts were oriented as closely as possible and all had the same background color. Alas, I'm sort of principal-componented-out, and am eager to get back to writing my blog post about "the only correct way to interview engineers", the punch-line of which is that you should only ask questions that involve golf balls, piano tuners, counterfeit coins, airplanes, treadmills, or piano tuners.

And so we leave things until part 3, "Shirt Language Processing", which will be forthcoming at some point after I muster up the motivation to either transcribe the shirt images or find an intern to do it for me.

T-Shirts, Feminism, Parenting, and Data Science, Part 1: Colors

2013-06-19T06:20:00+02:00

Before I was a parent I never gave much thought to children's clothing, other than to covet a few of the baby shirts at T-Shirt Hell. Now that I have a two-year-old daughter, I have trouble thinking of anything but children's clothing. (Don't tell my boss!)

What I have discovered over the last couple of years, is that clothing intended for boys is fun, whereas clothing intended for girls kind of sucks. There's nothing inherently two-year-old-boy-ish about dinosaurs, surfing ninjas, skateboarding globes, or "become-a-robot" solicitations, just as there's nothing inherently two-year-old-girl-ish about pastel-colored balloons, or cats wearing bows, or dogs wearing bows, or ruffles. Forget about gender, I want Madeline to grow up to be a "surfing ninja" kind of kid, not a "cats wearing bows" kind of kid. An "angry skateboarding dog" kind of kid, not a "shoes with pretty ribbons" kind of kid.

Accordingly, I have taken to buying all of Madeline's shirts in the boys section, the result of (her boy-ish haircut and) which is that half the time people refer to her as "he". This doesn't terribly bother me, especially if she ends up getting the gender wage premium that people are always yammering about on Facebook, but it makes me wonder why such a stark divide between toddler boy shirts and toddler girl shirts. And, of course, it makes me wonder if the divide is so stark that I can build a model to predict it!

The Dataset

I downloaded images of every "toddler boys" and "toddler girls" t-shirt from Carters, Children's Place, Crazy 8, Gap Kids, Gymboree, Old Navy, and Target. Because each one had their shirts at a different (random) website location, I decided that using an Image Downloader Chrome extension would be quicker and easier than writing a scraping script that worked with all the different sites.

I ended up with 616 images of boys shirts and 446 images of girls shirts. My lawyer has advised me against redistributing the dataset, although I might if you ask nicely.

Attempt #1: Colors

(As always, the code is on my GitHub.)

A quick glance at the shirts revealed that boys shirts tend toward boy-ish colors, girls shirts toward girl-ish colors. So a simple model could just take into account the colors in the image. I've never done much image processing before, so the Pillow Python library seemed like a friendly place to start. (In retrospect, a library that made at least a half-hearted attempt at documentation would probably have been friendlier.)

The PIL library has a getcolors function, that returns a list of

(# of pixels, (red, green, blue))

for each rgb color in the image. This gives 256 * 256 * 256 = almost 17 million possible colors, which is probably too many, so I quantized the colors by bucketing each of red, green, and blue into either [0,85), [85,170) or [170,255]. This gives 3 * 3 * 3 = 27 possible colors.

To make things even simpler, I only cared about whether an image contained at least one pixel of a given color [bucket] or whether it contained none. This allowed me to convert each image into an array of length 27 consisting only of 0's and 1's.

Finally, I trained a logistic regression model to predict, based solely on the presence or absence of the 27 colors, whether a shirt was a boys shirt or a girls shirt. Without getting too mathematical, we end up with a weight (positive or negative) for each of the 27 colors. Then for any shirt, we add up the weights for all the colors in the shirt, and if that total is positive, we predict "boys shirt", and if that total is negative, we predict "girls shirt".

I trained the model on 80% of the data and measured its performance on the other 20%. This (pretty stupid) model predicted correctly about 77% of the time.

Plotted below is the number of boys shirts (blue) and girls shirts (pink) in the test set by the score assigned them in the model. Without getting into gory details, a score of 0 means the model thinks it's equally likely to be a boys shirt or a girls shirt, with more positive scores meaning more likely boys shirt and more negative scores meaning more likely girls shirt. You can see that while there's a muddled middle, when the model is really confident (in either direction), it's always right.

If we dig into precision and recall, we see

P(is actually girl shirt | prediction is "girl shirt") = 75%\ P(is actually boy shirt | prediction is "boy shirt") = 77%\ P(prediction is "girl shirt" | is actually girl shirt) = 63%\ P(prediction is "boy shirt" | is actually boy shirt) = 86%

One way of interpreting the recall discrepancy is that it's much more likely for girls shirts to have "boy colors" than for boys shirts to have "girl colors", which indeed appears to be the case.

Superlatives

Given this model, we can identify

The Girliest Girls Shirt (no argument from me):

The Boyiest Girls Shirt (must be the black-and-white and lack of color?):

The Girliest Boys Shirt (I can see that if you just look at colors):

The Boyiest Boys Shirt (a slightly odd choice, but I guess those are all boy-ish colors?):

The Most Androgynous Shirt (this one is most likely some kind of image compression artifact, the main colors are boyish but the image also has some girlish purple pixels in it that cancel those out):

The Blandest Shirt (for sure!):

The Most Colorful Shirt (no argument with this one either!):

Scores for Colors

By looking at the coefficients of the model, we can see precisely which colors are the most "boyish" and which are the most "girlish". The results are not wholly unexpected:

	151.71
	80.68
	69.35
	49.69
	43.83
	40.99
	35.94
	30.56
	26.08
	24.06
	20.89
	20.49
	18.89
	17.67
	1.29
-17.37
-21.77
-29.95
-49.91
-56.4
-66.77
-69.52
-70.15
-82.17
-119.1
-175.2
-224.74

In Conclusion

In conclusion, by looking only at which of 27 colors are present in a toddler t-shirt, we can do a surprisingly good job of predicting whether it's a boys shirt or a girls shirt. And that pretty clearly involves throwing away lots of information. What if we were to take more of the actual image into account?

Coming soon, Part 2: EIGENSHIRTS