Work/PopChat

Discontinued · 20172016 — 2017

From inception
to 25M messages.

I was a founding engineer on PopChat — a Yahoo chatbot built from scratch for Kik and Facebook Messenger. Ten months later it was ranked #2 on Kik's bot shop, handling 25M+ messages, peaking at ~500 requests / minute, and proxying 200k+ messages between players. I also trained the custom 81-point facial landmark model that powered its face-effect features.

Featured in Yahoo's official blog →

RoleFounding Engineer

PlatformsKik · Facebook Messenger

BackendScala · Akka · Play

ML / CVPython · dlib · TensorFlow

Tenure~10 months (Aug 2016 — Jun 2017)

01 / CONTEXT

Why this project existed

Fun first. Assistance later.

The bigger goal was assistance on chat platforms — but selling users on that directly was hard.

Yahoo was exploring how to provide assistance to users on chat platforms — Kik and Facebook Messenger were the target audiences. Other Yahoo teams built bots that approached this directly: calendaring, scheduling, search-style helpers. Our team's theory was that this would be a hard sell. People don't go to chat platforms looking for a personal assistant — at least not at the scale we needed to validate the bet.

PopChat took the inverse angle: lead with fun, grow the user base first, then pivot toward assistance once we had the audience. The pivot landed on image manipulation — send PopChat a photo and ask it to put a hat on someone's head, swap a face with an emoji, or change the background. That's where the 81-point facial landmark model came in: it enabled precise placement around the head, including the forehead, which the standard 68-point model couldn't do.

02 / ROLE

Role & responsibilities

What I actually owned.

A 10-month founding-engineer engagement, split between Scala / Akka backend work, image ML in Python, and the cross-platform launch story across Kik and Facebook Messenger.

Founding engineer

There from inception. Built core game state machines, the multi-player matchmaking flow, the @mentions infrastructure refactor that turned a 1-on-1 bot into a group-chat platform, and the storage / auth integration with the in-house key-value, certificate, and object stores.

Scala · AkkaState machines@mentions refactorEnd-to-end

Image ML / CV

Designed and trained the custom 81-point facial landmark model that powered PopChat’s face-effect features (face-swap, emoji overlays). Built and forked the open-source eos library to extract 13 forehead landmarks from the Surrey Face Model, then trained a dlib shape predictor on the full 81-point dataset.

PythondlibTensorFlow81 landmarks

Scaling + reliability

Production scaling was a team effort — together we tuned Akka clustering, doubled the message frame size as load patterns shifted, and adopted in-house messaging proxies. Individual contribution on this front: a code-coverage push during a critical period (67.62% → 70.33%) and authoring several of the underlying scaling PRs.

Team effortAkka clusteringCode-coverage push

03 / SHIPPED

Selected work

Four blocks, ten months.

The work that defined PopChat across its short, dense lifetime — from the original trivia state machine through the @mentions pivot and the cross-platform launch. The 81-point ML model gets its own section below.

FEAT_01Foundational system · 2016

Trivia state machine + game flow

The original game system: Singleplayer, Multiplayer matchmaking, category selection, answer-timer states, and game-end / restart flows. The bedrock the rest of PopChat was built on.

Joined early in the state machine’s life and contributed heavily from there — added multiplayer matchmaking, category selection, restart / quit flows, and answer-time gating across dozens of state transitions.
Conversation state held in the in-house KV store, with per-user push / pop semantics for restart and quit flows.
Worked through dozens of edge cases: late joiners, timed-out players, match-error recovery, suggestion rotation.
Question-content pipeline: CSV → JSON conversion scripts, automated fetching, source-logging into the in-house log pipeline.

ScalaAkkaPlay frameworkKV store

FIG · 01

FEAT_02Architectural pivot · 2016

@mentions infrastructure refactor

Mid-project pivot: turn the 1-on-1 trivia bot into a group-chat platform where users could @mention the bot inside any conversation. Required a major infrastructure refactor and a new sharding model — conversation-id for @mentions, activity-id for direct chat.

Owned the “Major infrastructure refactor to prepare for @mentions” PR and its long tail of follow-on changes.
Per-user suggested-response rotation so two players in the same group saw different choices.
New flow states for late join, partial timeout, and group game-end — including their functional tests via the in-house test harness.
Re-keyed sharding: conversation-id for @mentions, activity-id for direct bot chat.

ScalaAkkaShardingPlay framework

PopChat group-chat menu showing @mention bot suggestions

FIG · 02

FEAT_03Reusable game engine · 2017

Mini-game platform expansion

Once the trivia foundation was solid, generalized the engine to support multiple games. TicTacEmoji, Drawstory, Magic8Ball, Storytime, and Faceswap all rode on top of the same state-machine + @mentions infrastructure.

Extracted shared game-flow primitives so each new game was a thin layer, not a rewrite.
Released TicTacEmoji, Drawstory, Magic8Ball, and Storytime in a single batch.
Faceswap consumed the in-house image-processing proxy — see the 81-point landmark section below.
Trivia content pipeline matured: 9+ categories with automated question fetching from QuestionCo.

ScalaAkkaImage-processing proxyProtobuf

FIG · 03

FEAT_04Cross-platform expansion · 2017

Facebook Messenger launch

After hitting #2 on Kik, ported PopChat to Facebook Messenger. The platforms had different UX rules — Facebook capped answer text length, lacked some of the suggestion affordances Kik had — so the launch wasn’t just a transport swap.

Trimmed all trivia answers to ≤20 characters for Facebook’s constraint.
Suppressed in-bot notifications on Facebook (platform policy mismatch).
Adjusted matchmaking + main-menu copy per platform conventions.
Featured in Yahoo’s official blog announcement on the bot launches.

ScalaAkkaFB Messenger API

04 / RECEIPTS

Measurable outcomes

Receipts.

Engineering and product numbers from PopChat’s launch period. Internal user-cohort metrics aren’t captured here.

25M+

Messages handled

Kik + Facebook · 2016 — 2017

Ranking on Kik’s bot shop

Bot shop overall · 2016 — 2017

~500/min

Peak request rate

Platform throughput at launch peak

200k+

Messages proxied between users

@mentions multi-player mode

10 mo

From inception through launch

Aug 2016 — Jun 2017 · End-to-end

492★

Open-source 81-landmark model

github.com/codeniko · cited in PhD-level papers

Facial landmarks (vs 68 default)

Custom dlib shape predictor

Mini-games shipped on the platform

Trivia · TicTacEmoji · Drawstory · Magic8Ball · Storytime · Faceswap

05 / ML

Open source · ML

81-point shape predictor.

PopChat needed face-effects that anchored objects on the top of users’ heads — but dlib’s 68-point landmark model has no forehead points. So I trained a custom shape predictor that adds 13 of them.

The technique: forked patrikhuber's eos library to access the Surrey Face Model triangle mesh, identified 13 specific points along the forehead I wanted, then modified eos to extract those coordinates per image in the ibug dataset. The output: each image now had 81 landmark coordinates instead of 68. Trained the resulting dataset with dlib'strain_shape_predictorto produce the final 81-point model.

Why it matters: forehead landmarks are exactly what you need for features like “put a hat on this user's head.” The model has had a long tail: 492★ / 126 forks on GitHub, and a handful of PhD-level papers have cited or used it as a baseline when comparing landmark approaches. The repo itself is no longer actively maintained.

Model on GitHub eos fork

GITHUB · OPEN SOURCE● PUBLIC · ARCHIVED

Repocodeniko/shape_predictor_81_face_landmarks

Stars492★

Forks126

Landmarks68 → 81 (added 13 forehead points)

Trained onibug face dataset · Surrey Face Model

StackPython · dlib · eos · NumPy · OpenCV

Cited in5+ PhD-level papers (comparison / baseline) — links forthcoming

Used byPopChat face-effects, plus 126 community forks

# Personal open-source repo. Released 2017; no longer actively maintained, but the work has had a long tail in academic and community use.

06 / STACK

The technical environment

What I worked in, daily.

A snapshot of the stack PopChat ran on. Internal-only auth specifics, sharding details, and protobuf schemas are not described here.

Languages

$ Scala (primary)
$ Python (ML)

Server

$ Akka clustering
$ Play framework
$ Akka HTTP
$ Protobuf

Storage & infra

$ In-house KV store
$ In-house auth / certs
$ In-house object store

ML / CV

$ dlib
$ TensorFlow
$ eos (forked)
$ Surrey Face Model

Testing & quality

$ ScalaTest
$ scalastyle
$ In-house bot test harness

07 / DIFFERENTLY

Honest retrospectives

What I’d do differently.

My voice, my opinions. PopChat was eight years ago — these are the takeaways that have stuck with me.

Bot platforms shifted faster than the bots could adapt

Kik’s bot shop lost momentum within months of our launch and the platform itself wound down later. Building heavily around one platform’s UX rules (Kik’s emoji + suggestion-card affordances) made the Facebook Messenger port more painful than it should have been. If I were doing this again, I’d treat platform-specific UX as a thinner adapter layer from day one.

Underweighted content automation upfront

For the first months we hand-curated trivia questions per category. Once we automated fetching from a content provider and added Splunk source-logging, the maintenance cost dropped sharply. The right move was to build that pipeline in week one, not month four.

Akka tuning was reactive, not proactive

Doubled the Akka frame size only after seeing real production load issues. A pre-launch load test and a single afternoon of Akka tuning would have prevented the live incident that prompted it.

PREVIOUS CASE STUDY

Yahoo Mail

Yahoo Mail for Android

From inceptionto 25M messages.