Filtering noisy game telemetry and app events

Heartbeats, per-frame game telemetry and events from QA accounts are events nobody reads row by row, and without a filter they reach every destination. A Jitsu function drops them before any destination sees them and keeps a stable 1-in-N sample of performance telemetry. Dropped events are not billed unless the function called fetch before dropping them.

TL;DR

• drop-noise drops heartbeats, frame ticks and QA accounts, and keeps performance samples from 1 in N players.
• Dropped events are not billed unless the function called fetch before dropping them.
• An event that reaches any destination counts once, so the filter goes on every connection.
• Warehouse connections deduplicate resent copies on message_id. ClickHouse needs FINAL for exact counts.
• Purchases, sign-ups and anything else you count or sum stay unsampled.

Who this is for

The engineer who owns the analytics bill or the warehouse, and the client engineer who decides what the app sends. Changing the filter needs no app release, because it runs in Jitsu.

The problem

Apps and games send four kinds of noise:

• Keep-alive events. A heartbeat every few seconds keeps presence or session logic alive in the app.
• Per-frame or per-tick telemetry. One event per frame multiplies volume by the frame rate. Aggregate on the device into a performance_sample every 60 seconds, then sample that.
• QA and debug traffic. Internal testers replay the same flows and skew the funnels they touch.
• Resent batches. A mobile network drops the response, the library resends the batch, and the same messageId arrives twice. Jitsu's deduplication docs name mobile networks as a cause.

Measure before you drop anything. The first query below shows which events make up the volume.

Noise events and what the function does with them

The full plan with properties is in the tracking plan CSV.

How events get in

Nothing changes on the device. Apps keep sending through Segment's Analytics-Swift, Analytics-Kotlin or analytics-react-native, pointed at your Jitsu domain (iOS, Android), and game clients keep posting to the HTTP API. The filter is a function attached to each connection.

The function

drop-noise is TypeScript in a jitsu-cli project, tested with gameplay events, heartbeats, QA accounts, a 2,000-player sampling run and malformed input:

import { JitsuFunction } from "@jitsu/protocols/functions";
import { AnalyticsServerEvent } from "@jitsu/protocols/analytics";

export const config = {
  slug: "drop-noise",
  name: "Drop and sample telemetry",
  description: "Drops heartbeats, frame ticks and QA accounts. Keeps performance samples from 1 in N players.",
};

// Events nobody reads one row at a time. Returning "drop" keeps them out of every destination.
const DROP_EVENTS = new Set(["heartbeat", "frame_tick"]);

// FNV-1a (32-bit). Stable across runs, so the same player always lands in the same bucket.
function bucketOf(id: string, buckets: number): number {
  let h = 0x811c9dc5;
  for (let i = 0; i < id.length; i++) {
    h ^= id.charCodeAt(i);
    h = Math.imul(h, 0x01000193);
  }
  return (h >>> 0) % buckets;
}

const dropNoise: JitsuFunction<AnalyticsServerEvent> = async event => {
  const name = event.type === "track" && typeof event.event === "string" ? event.event : "";
  if (DROP_EVENTS.has(name)) {
    return "drop";
  }

  // Comma-separated user IDs of internal test accounts. Set QA_USER_IDS on the connection.
  const qaUsers = (process.env.QA_USER_IDS ?? "").split(",").map(id => id.trim()).filter(Boolean);
  if (event.userId && qaUsers.includes(String(event.userId))) {
    return "drop";
  }

  if (name === "performance_sample") {
    // Keep 1 in PERF_SAMPLE_EVERY players (default 10). A kept player keeps every sample.
    const every = Math.max(1, Math.floor(Number(process.env.PERF_SAMPLE_EVERY) || 10));
    const player = event.anonymousId || event.userId;
    if (!player || bucketOf(String(player), every) !== 0) {
      return "drop";
    }
    return { ...event, properties: { ...(event.properties ?? {}), sample_rate: every } };
  }

  return event;
};

export default dropNoise;

This code is built and deployed with jitsu-cli; the function editor in the Jitsu UI takes JavaScript, which is what jitsu-cli build outputs.

Set two environment variables on the connection: QA_USER_IDS, a comma-separated list of internal account IDs, and PERF_SAMPLE_EVERY, the sampling rate, which defaults to 10. Functions read them through process.env (runtime docs). The bucket hash is a few lines of plain JavaScript (FNV-1a), so the function has no imports beyond types.

Sampling is per player, not per event. A kept player keeps every sample, so frame-rate distributions for that player stay intact, and sample_rate on each kept row lets SQL scale counts back up.

QA_USER_IDS catches testers after they sign in. Give internal builds their own Jitsu site and write key to keep all of their events out of production destinations.

Resent copies

Jitsu's deduplication is a feature of warehouse connections: warehouses keep one row per message_id, and ClickHouse needs FINAL for exact counts.

The billing rule

Jitsu bills active events: an event successfully sent to at least one destination, counted once however many it reaches. Captured events are free.

Dropped events are not billed unless the function called fetch before dropping them, and drop-noise does not call fetch.

A function that splits one event into several is billed per resulting event (pricing).

Placement decides whether a drop saves anything. A heartbeat dropped on the Mixpanel connection but delivered to ClickHouse is still one active event, because it reached a destination. An event goes unbilled only when every connection drops it and no function called fetch before the drop, so attach drop-noise to every connection on the stream.

What not to drop

• Anything you count or sum. Purchases, subscriptions, sign-ups and level_end stay unsampled.
• First-session events. Application Installed, onboarding steps and the first Application Opened define retention cohorts.
• Errors. client_error is rare and each one is worth reading.
• Events you will want later. A dropped event is gone from every connection that dropped it. To keep a raw copy, leave the filter off an Amazon S3 or Google Cloud Storage connection, and include those events in your volume estimate.

Heartbeats deserve one more check. The sessions query ends a session after 30 minutes without an event. Once heartbeats are gone, a session runs from the first to the last remaining event.

The query

ClickHouse, against the default events table. Run this first, before the function, to see where the volume is:

-- Which track events make up the volume. Run it before deciding what to drop.
SELECT
    event,
    count() AS events,
    round(count() / sum(count()) OVER (), 3) AS share_of_track_events,
    uniqExact(anonymous_id) AS installs_sending_it
FROM events FINAL
WHERE type = 'track' AND timestamp >= today() - 7
GROUP BY event
ORDER BY events DESC
LIMIT 20;

After sampling, read performance data back with the rate applied:

-- Frame rate by device model from a 1-in-N player sample.
-- sum(sample_rate) estimates how many samples the full player base sent.
SELECT
    device_model,
    sum(sample_rate) AS estimated_samples,
    uniqExact(anonymous_id) AS sampled_installs,
    quantileExact(0.5)(fps_p10) AS median_fps_p10
FROM events FINAL
WHERE type = 'track' AND event = 'performance_sample' AND timestamp >= today() - 7
GROUP BY device_model
ORDER BY estimated_samples DESC
LIMIT 20;

Only sampled players feed median_fps_p10, while estimated_samples multiplies each kept row by its sample_rate.

Destinations

Attach drop-noise to every connection: ClickHouse or another warehouse, Mixpanel, Amplitude, PostHog, Braze, Statsig and Webhook. Warehouse connections deduplicate on message_id.

Example implementation (illustrative): a racing game with telemetry

Illustrative example: not a Jitsu customer. A mobile racing game sends a heartbeat every few seconds for its online lobby, a performance sample every 60 seconds of racing, and race results from its game server.

Its stream feeds ClickHouse, Amplitude, and a Webhook that posts to an internal anti-cheat service. drop-noise sits on all three connections, with the QA team's account IDs in QA_USER_IDS.

What others in the industry do

These write-ups describe other companies' own pipelines. None of these companies is a Jitsu customer:

• MY.GAMES describes clients resending batches over poor mobile connections.
• Supercell's account of leaving a packaged web analytics tool lists sampling, sometimes down to 1% of events, as a cause of incomplete data.

Where Jitsu does not fit

• Aggregation. Functions act on one event at a time. Rolling per-frame data into one sample happens on the device, and rollups across players happen in the warehouse.
• Restoring dropped events. Jitsu Cloud keeps a raw backup of every accepted event (7 days on Free, up to 90 days on paid plans); restores are done by the Jitsu team, and self-hosted Jitsu keeps no backup (event backups).
• Volume you keep. If most of your volume is telemetry you choose to keep, per-event billing grows with it, and filtering does not change that.

Questions people ask

Are events dropped in a function billed? No, unless the function called fetch before dropping them. An event that reaches at least one destination counts once as an active event, so drop an event on every connection to keep it off the bill.

Does sampling by player bias frame-rate numbers? Not for distributions, because each kept player keeps every sample. Counts are scaled back with sample_rate. Never sample events you count or sum, such as purchases or level completions.

Should debug builds send to the same Jitsu site? No. Give internal builds their own site and write key, so their events never reach production destinations.

Get started with Jitsu

Get started on Jitsu Cloud: the Free plan includes 200k active events a month.

Download the tracking plan, no form. To talk through volume or self-hosting, book a demo. The overview covers the whole pipeline.