{"id":1864,"date":"2026-06-29T13:36:43","date_gmt":"2026-06-29T06:36:43","guid":{"rendered":"http:\/\/108.136.222.107\/?post_type=tmi_blog&p=1864"},"modified":"2026-06-30T14:22:25","modified_gmt":"2026-06-30T07:22:25","slug":"test-blog-4","status":"publish","type":"tmi_blog","link":"https:\/\/www.tokaicom-mitra.co.id\/jp\/blog\/test-blog-4\/","title":{"rendered":"Jumpy Hands"},"content":{"rendered":"

Building a Gesture-Controlled Serverless Game on AWS<\/strong><\/h2>\n\n\n\n

How we built Jumpy Hands \u2014 a browser-based game controlled entirely by hand gestures \u2014 using MediaPipe, HTML5 Canvas, and a fully serverless AWS backend.<\/p>\n\n\n\n

The Idea<\/h3>\n\n\n\n

We wanted to build something interactive for an AWS Summit booth \u2014 something that would catch people’s attention, get them to stop and play, and showcase the power of serverless AWS services in a tangible way.<\/p>\n\n\n\n

The result: **Jumpy Hands**, a Flappy Bird-style game where players use their webcam to control the game character with hand gestures. Open hand to glide, clench a fist to jump. No keyboard, no controller \u2014 just your hands.<\/p>\n\n\n\n

How it works<\/h3>\n\n\n\n

Gesture detection<\/h4>\n\n\n\n

The core interaction relies on **MediaPipe Hands** by Google, running entirely in the browser. Here’s the detection pipeline:<\/p>\n\n\n\n

1. The webcam feed is captured at 320×240 resolution (optimized for performance)<\/p>\n\n\n\n

2. MediaPipe processes each frame and returns 21 hand landmarks per detected hand<\/p>\n\n\n\n

3. Our gesture classifier analyzes the landmarks to determine: **open hand** or **closed fist**<\/p>\n\n\n\n

The gesture detection logic is straightforward \u2014 we compare fingertip positions against the proximal interphalangeal (PIP) joint positions:<\/p>\n\n\n\n

```javascript```\ndetectGesture(landmarks) {\n    const fingerTips = [8, 12, 16, 20];\n    const fingerPIPs = [6, 10, 14, 18];\n\n    let curledFingers = 0;\n    for (let i = 0; i < fingerTips.length; i++) {\n        const tip = landmarks[fingerTips[i]];\n        const pip = landmarks[fingerPIPs[i]];\n        if (tip.y > pip.y) {\n            curledFingers++;\n        }\n    }\n\n    \/\/ Thumb check (different axis)\n    const thumbTip = landmarks[4];\n    const thumbIP = landmarks[3];\n    const indexMCP = landmarks[5];\n    if (Math.abs(thumbTip.x - indexMCP.x) < Math.abs(thumbIP.x - indexMCP.x)) {\n        curledFingers++;\n    }\n\n    return (curledFingers \/ 5) >= 0.6 ? 'fist' : 'open';\n}<\/code><\/pre>\n\n\n\n
If 60% or more of the fingers are curled, we classify it as a fist. The fist-to-open transition triggers a jump. This simple threshold approach proved reliable enough for a game setting.<\/p>\n\n\n\n
Two-Player Mode<\/strong><\/h4>\n\n\n\n
In two-player mode, both players share a single webcam. We detect up to two hands and assign them to players based on their horizontal position in the frame \u2014 left half belongs to Player 1, right half to Player 2. The webcam image is mirrored so it feels natural.<\/p>\n\n\n\n
Both players get the same pipe sequence (using a seeded random number generator) to keep it fair:<\/p>\n\n\n\n
```javascript```\n\/\/ Simple seeded RNG (mulberry32)\ncreateRNG(seed) {\n    return function() {\n        let t = seed += 0x6D2B79F5;\n        t = Math.imul(t ^ t >>> 15, t | 1);\n        t ^= t + Math.imul(t ^ t >>> 7, t | 61);\n        return ((t ^ t >>> 14) >>> 0) \/ 4294967296;\n    };\n}<\/code><\/pre>\n\n\n\n
Game Rendering<\/strong><\/h4>\n\n\n\n
The game itself is rendered on an HTML5 Canvas at 60fps. No game engine \u2014 just vanilla JavaScript with a standard game loop:<\/p>\n\n\n\n
– Bird physics: gravity + velocity + delta-time normalization<\/p>\n\n\n\n
– Pipe spawning: time-interval based, positions generated from the seeded RNG<\/p>\n\n\n\n
– Collision detection: axis-aligned bounding box check against pipe gaps<\/p>\n\n\n\n
– Visual feedback: the webcam border changes color (red = no hand, green = open, yellow = fist)<\/p>\n\n\n\n
The Architecture<\/strong><\/h3>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The entire backend is serverless. No EC2 instances, no containers \u2014 just event-driven functions that scale to zero when idle.<\/p>\n\n\n\n
AWS Services Deep Dive<\/strong><\/h3>\n\n\n\n
AWS Amplify \u2014 Frontend Hosting<\/strong><\/h4>\n\n\n\n
The static frontend (HTML, CSS, JS, assets) is hosted on AWS Amplify. It auto-deploys on every git push to main \u2014 no manual build\/deploy steps. Under the hood, Amplify provisions an S3 bucket and CloudFront CDN distribution, giving us HTTPS (required for webcam access) and global edge delivery.<\/p>\n\n\n\n
Amazon API Gateway \u2014 REST API<\/strong><\/h4>\n\n\n\n
Three endpoints exposed to the browser:<\/p>\n\n\n\n
Method<\/td>Path<\/td>Purpose<\/td><\/tr>
POST<\/td>\/scores<\/td>Submit a game score<\/td><\/tr>
GET<\/td>\/leaderboard<\/td>Fetch top scores<\/td><\/tr>
GET<\/td>\/analytics<\/td>Get aggregate stats<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
CORS is enabled for browser access, and each endpoint routes to a dedicated Lambda function.<\/p>\n\n\n\n
AWS Lambda \u2014 Backend Logic<\/strong><\/h4>\n\n\n\n
Six Lambda functions (Python 3.12, 256MB memory, 10s timeout):<\/p>\n\n\n\n
– **SubmitScore** \u2014 Receives a score from the frontend and kicks off the Step Functions workflow<\/p>\n\n\n\n
– **GetLeaderboard** \u2014 Queries DynamoDB’s Global Secondary Index for top scores, sorted descending<\/p>\n\n\n\n
– **GetAnalytics** \u2014 Returns aggregate metrics (total games, averages, medians)<\/p>\n\n\n\n
– **SaveScore** \u2014 Step Function task that persists the score to DynamoDB<\/p>\n\n\n\n
– **CheckRanking** \u2014 Step Function task that determines if the score cracks the top 10<\/p>\n\n\n\n
– **UpdateLeaderboard** \u2014 Step Function task that pushes the updated leaderboard to IoT Core<\/p>\n\n\n\n
Amazon DynamoDB \u2014 Score Storage<\/strong><\/h4>\n\n\n\n
Single table design with a partition key (`pk`) and sort key (`sk`). A Global Secondary Index on the `score` field enables efficient top-N queries without scanning.<\/p>\n\n\n\n
```yaml```\nBillingMode: PAY_PER_REQUEST\nGlobalSecondaryIndexes:\n  - IndexName: score-index\n    KeySchema:\n      - AttributeName: pk\n        KeyType: HASH\n      - AttributeName: score\n        KeyType: RANGE<\/code><\/pre>\n\n\n\n
PAY_PER_REQUEST billing means we pay nothing when the game isn’t being played.<\/p>\n\n\n\n
AWS Step Functions \u2014 Post-Game Orchestration<\/strong><\/h4>\n\n\n\n
When a player finishes a game, the submission doesn’t just write to a database \u2014 it triggers a workflow:<\/p>\n\n\n\n
```json```\nSaveScore \u2192 CheckRanking \u2192 (if top 10?) \u2192 UpdateLeaderboard<\/code><\/pre>\n\n\n\n
This decouples the API response from the downstream processing. The player gets an immediate “score submitted” response, while Step Functions handles the rest asynchronously. If the score ranks in the top 10, the leaderboard update is pushed to all connected clients.<\/p>\n\n\n\n
AWS IoT Core \u2014 Real-Time Leaderboard<\/strong><\/h4>\n\n\n\n
IoT Core is used purely for its MQTT pub\/sub messaging over WebSocket \u2014 no IoT “things” or rules involved.<\/p>\n\n\n\n
– **Topic:** `flappy-bird\/leaderboard`<\/p>\n\n\n\n
– **Publisher:** The UpdateLeaderboard Lambda function<\/p>\n\n\n\n
– **Subscribers:** Any browser client connected via MQTT over WebSocket<\/p>\n\n\n\n
When a new high score lands, every connected browser instantly receives the updated leaderboard without polling.<\/p>\n\n\n\n
Amazon Cognito \u2014 Authentication<\/strong><\/h4>\n\n\n\n
The Identity Pool allows **unauthenticated (guest) identities** \u2014 players don’t need to sign up or log in. Cognito issues temporary IAM credentials scoped to:<\/p>\n\n\n\n
– `iot:Connect` \u2014 establish WebSocket connection<\/p>\n\n\n\n
– `iot:Subscribe` \u2014 subscribe to the leaderboard topic<\/p>\n\n\n\n
– `iot:Receive` \u2014 receive messages<\/p>\n\n\n\n
This keeps the experience frictionless while maintaining secure access to IoT Core.<\/p>\n\n\n\n
Amazon QuickSight \u2014 Live Analytics (Optional)<\/strong><\/h4>\n\n\n\n
For the Summit booth, we added a secondary display showing live game analytics powered by QuickSight with Amazon Q:<\/p>\n\n\n\n
– Score distributions, top players, games over time<\/p>\n\n\n\n
– Natural language queries: “Who has the highest score?” or “What’s the average score in two-player mode?”<\/p>\n\n\n\n
QuickSight connects to the score data via an S3 export (triggered by DynamoDB Streams) with Athena as an intermediary for Direct Query mode.<\/p>\n\n\n\n
Infrastructure as Code<\/strong><\/h3>\n\n\n\n
The entire backend is defined in a single AWS SAM template (`template.yaml`). Deploying the full stack takes two commands:<\/p>\n\n\n\n
```bash```\nsam build\nsam deploy<\/code><\/pre>\n\n\n\n
SAM outputs the API URL, Cognito IDs, and other values needed for the frontend config. The frontend just needs those values in `config.js` and it’s connected.<\/p>\n\n\n\n
Performance Considerations<\/strong><\/h3>\n\n\n\n
The GPU Problem<\/strong><\/h4>\n\n\n\n
MediaPipe’s hand tracking model runs on WebGL and is computationally demanding. On laptops with weak integrated graphics, frame rates drop and gesture detection becomes laggy or unresponsive.<\/p>\n\n\n\n
We mitigated this by:<\/p>\n\n\n\n
– Running the webcam at 320×240 instead of higher resolutions<\/p>\n\n\n\n
– Using `modelComplexity: 0` (fastest MediaPipe model)<\/p>\n\n\n\n
– Lowering `minTrackingConfidence` to 0.3 to reduce re-detection frequency<\/p>\n\n\n\n
– Yielding between inference frames (`setTimeout(resolve, 10)`) to let the game loop breathe<\/p>\n\n\n\n
For hardware that still struggles, we built **Jumpy Hands Lite** \u2014 a separate build using TensorFlow.js Hand Pose Detection with the WebGL backend, which is lighter on GPU resources.<\/p>\n\n\n\n
Delta-Time Normalization<\/strong><\/h4>\n\n\n\n
The game loop uses delta-time normalization to maintain consistent physics regardless of frame rate:<\/p>\n\n\n\n
```javascript```\nconst rawDelta = now - this.lastTime;\nconst delta = Math.min(rawDelta, 50) \/ 16.67; \/\/ 1.0 = 60fps\n\nthis.bird.velocity += GRAVITY * delta;\nthis.bird.y += this.bird.velocity * delta;<\/code><\/pre>\n\n\n\n
The `Math.min(rawDelta, 50)` clamp prevents the bird from teleporting if the tab was hidden or a frame took too long.<\/p>\n\n\n\n
Cost<\/strong><\/h3>\n\n\n\n
Service<\/td>Monthly Cost<\/td><\/tr>
Amplify, Lambda, DynamoDB, API Gateway, Cognito, S3<\/td>Free tier<\/td><\/tr>
IoT Core<\/td>~$0.01<\/td><\/tr>
QuickSight (Enterprise, 1 author)<\/td>~$24<\/td><\/tr>
**Total**<\/td>**~$24**<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Everything except QuickSight falls within AWS Free Tier for demo-level traffic. The architecture scales to zero \u2014 if nobody plays for a month, you pay almost nothing.<\/p>\n\n\n\n
Lessons Learned<\/strong><\/h3>\n\n\n\n
1. **MediaPipe is impressive but GPU-hungry.** Test on the weakest hardware your audience might have. At a booth, you control the hardware \u2014 but for a web game distributed broadly, GPU requirements are a real barrier.<\/p>\n\n\n\n
2. **IoT Core is underrated for web real-time.** We initially considered AppSync or WebSocket API Gateway, but IoT Core’s MQTT pub\/sub with Cognito credentials was simpler to implement and more cost-effective for one-directional pushes.<\/p>\n\n\n\n
3. **Step Functions add clarity, not just orchestration.** The post-game workflow could have been a single Lambda, but breaking it into discrete steps made it trivial to debug, monitor, and extend later.<\/p>\n\n\n\n
4. **Seeded RNG is essential for fairness.** In two-player mode, both players see identical pipe sequences. Without a shared seed, one player could get an easier run purely by chance.<\/p>\n\n\n\n
5. **HTTPS is non-negotiable for webcam access.** Browsers block `getUserMedia()` on plain HTTP. Amplify handles this automatically, but it’s easy to forget during local development (`localhost` is the exception).<\/p>","protected":false},"excerpt":{"rendered":"
Building a Gesture-Controlled Serverless Game on AWS How we built Jumpy Hands \u2014 a browser-based game controlled entirely by hand gestures \u2014 using MediaPipe, HTML5 Canvas, and a fully serverless AWS backend. The Idea We wanted to build something interactive for an AWS Summit booth \u2014 something that would catch people’s attention, get them to stop and play, and showcase the power of serverless AWS services in a tangible way. The result: **Jumpy Hands**, a Flappy Bird-style game where players use their webcam to control the game character with hand gestures. Open hand to glide, clench a fist to jump. No keyboard, no controller \u2014 just your hands. How it […]<\/p>\n","protected":false},"author":2,"featured_media":1837,"template":"","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}}},"blog_category":[15,21,22],"blog_tag":[45,44,35,41,38,43],"class_list":["post-1864","tmi_blog","type-tmi_blog","status-publish","has-post-thumbnail","hentry","blog_category-ai","blog_category-quick","blog_category-serverless","blog_tag-apigateway","blog_tag-cognito","blog_tag-amplify","blog_tag-cloudfront","blog_tag-lambda","blog_tag-stepfunctions"],"_links":{"self":[{"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/tmi_blog\/1864","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/tmi_blog"}],"about":[{"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/types\/tmi_blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/users\/2"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/media\/1837"}],"wp:attachment":[{"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/media?parent=1864"}],"wp:term":[{"taxonomy":"blog_category","embeddable":true,"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/blog_category?post=1864"},{"taxonomy":"blog_tag","embeddable":true,"href":"https:\/\/www.tokaicom-mitra.co.id\/jp\/wp-json\/wp\/v2\/blog_tag?post=1864"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}