--- title: "Data Transfer: Consumer" slug: "nodejs-data-transfer-consumer" description: "How the Node.js Library sends tracking requests: queue, batches, retries and discarded batches for the HTTP_CLIENT, FORK_CURL, FILE and CUSTOM consumers." status: "update" updated: 2026-11-03T12:06:16Z published: 2026-11-03T12:06:16Z canonical: "docs.mapp.com/nodejs-data-transfer-consumer" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.mapp.com/llms.txt > Use this file to discover all available pages before exploring further. # Data Transfer: Consumer Consumers normally send tracking requests to the Intelligence tracking server. The options to do this are: - **HTTP_CLIENT**: Saves tracking requests in memory and sends them to the server using the same process as the library. - **FORK_CURL**: Saves tracking requests in memory and sends them in a separate, forked process to the server​. - **FILE**: Best for high traffic websites. Requests are saved in a separate file and sent via a cronjob to the tracking server in a specified interval. - **CUSTOM**: Create your own consumer to send the tracking requests. All consumers are fed by the request queue of the tracking instance. Adding a request does not send it: the queue is sent when `flush()` is called, every `requestInterval` seconds if an interval is configured, when the queue reaches `maxQueueSize` and when the tracking instance is closed (see [Classes](/v1/docs/nodejs-classes)). A batch contains at most `maxBatchSize` requests and is closed at about 20 MB request body. What happens with a batch is decided by the consumer: | Result | Meaning | | --- | --- | | `SENT` | The batch was delivered and leaves the queue. | | `RETRY` | The batch could not be delivered right now. It goes back to the front of the queue and is retried after `attemptTimeout` milliseconds, up to `maxAttempt` attempts per flush. After the last attempt the requests **stay in the queue** and are sent again with the next flush (or when the instance is closed). | | `DISCARDED` | The batch can never be sent (for example a permanent client error). It leaves the queue, so it cannot block the requests behind it. The flush reports `false`. The built-in consumers log it at level ERROR; for a `DISCARDED` batch of a custom consumer the library writes no log entry. | > [!NOTE] > Note > > Requests that are still in the queue when the tracking instance is closed get one last flush with up to `maxAttempt` attempts, within the 30-second deadline of `close()`. Whatever is still undelivered after that is lost. When the queue reaches `maxQueueSize`, it is sent; if that fails, the oldest requests are discarded (logged at level WARN), at least `maxBatchSize` of them, so the next attempt is made only after that many new requests. `0` means unlimited. **HTTP_CLIENT** The HTTP_CLIENT consumer is best for low-traffic websites or when using the library only to track specific pages. It ensures a fast responses and processing time for the track requests. For higher traffic websites (more than 20 requests per second), it is better to use the FILE transfer consumer. > [!NOTE] > Note > > If you send browser requests via pixel to the track server directly and only the important requests via the HTTP_CLIENT consumer, send them right away: create one tracking instance per request and `await close()` it at the end of the request, or call `await flush()` after `track()` (a long-lived instance can also use `setRequestInterval()`). Since version 1.0.0, a max batch size of 1 no longer sends a request immediately, it only limits the number of requests per batch. By default, the attempt timeout is set to 100 milliseconds, with a maximum batch size of 50 requests, one attempt per flush (`maxAttempt` 1), a maximum queue size of 1000 requests and no automatic interval (`requestInterval` 0). The HTTP_CLIENT consumer sends each batch as one `POST` request to `https://<trackDomain>/<trackId>/batch`, using the configured `connectionTimeout` (default 30 s) and `readTimeout` (default 5 s). The read timeout is an inactivity timeout: it ends the request if no data arrives for that time, not after a fixed total time. Connections are reused: the response is read completely and the connection goes back to a keep-alive agent that all tracking instances of the process share, so consecutive batches do not need a new TCP connection and TLS handshake each. How long an idle connection stays open is decided by the keep-alive timeout of the server; an idle connection does not keep the process alive, and a connection with an error or a timeout is never reused. The response decides what happens with the batch: | Response | Result | | --- | --- | | Any status below 400 | `SENT` | | 5xx, 408 (Request Timeout), 429 (Too Many Requests) | `RETRY` | | Any other 4xx (e.g. 400, 404, 413) | `DISCARDED`, logged at error level | | Network error (connection refused, timeout, ...) | `RETRY` | | Batch with more than 10,000 requests or a payload of 24 MB or more | `DISCARDED`, logged at error level | | Empty payload | `RETRY` | > [!NOTE] > Note > > A permanent client error (for example caused by a wrong track ID) discards the affected batches instead of retrying them forever. Watch the error log when setting up a new configuration. The limits are measured in bytes (1 MB = 1024 × 1024). ```javascript const config = new MappIntelligenceConfig()    .setTrackDomain("analytics01.wt-eu02.net")    .setTrackId("111111111111111")    .setConsumerType(MappIntelligenceConsumerType.HTTP_CLIENT)    .setMaxAttempt(1)    .setAttemptTimeout(100)    .setMaxBatchSize(50)    .setMaxQueueSize(1000)    .setRequestInterval(0); ``` **FORK_CURL** The FORK_CURL consumer works with the curl cli and uses a separate process to send them in batches to the tracking server. This can optimize balancing load distribution on the server. FORK_CURL is best to use on low-traffic websites with up to 20 requests per second. curl is started without a shell, and the payload is passed via stdin: nothing in a request is interpreted, the requests (with IP addresses and user agents) do not show up in the process list or in the debug log, and the size of a batch is not limited by the maximum length of a command line. The library waits for curl to finish, the responses are handled like with the HTTP_CLIENT consumer. If curl is not installed, the result is `RETRY`. curl gets `--connect-timeout` from `connectionTimeout` (if it is above 0) and the maximum time of the request `-m` as connection + read timeout, but only if both timeouts are above 0. > [!NOTE] > Note > > The same applies to the FORK_CURL consumer: a max batch size of 1 does not send a request immediately. Use `close()` at the end of the request, `flush()` or `setRequestInterval()`. By default, the attempt timeout is set to 100 milliseconds, with a maximum batch size of 50 requests, and a maximum queue size of 1000 requests. ```javascript const config = new MappIntelligenceConfig()    .setTrackDomain("analytics01.wt-eu02.net")    .setTrackId("111111111111111")    .setConsumerType(MappIntelligenceConsumerType.FORK_CURL)    .setMaxAttempt(1)    .setAttemptTimeout(100)    .setMaxBatchSize(50)    .setMaxQueueSize(1000); ``` **FILE** For high traffic websites, it's best to use a FILE transfer. This is because it helps to reduce the load and memory used by your server. Instead of saving the requests in your memory, it writes the requests in a file and saves it there before sending it in batches to the track server. This solution is also more reliable in data collection: if sending a log file fails, the cron job keeps the file and sends it again in its next run, while the *HTTP_CLIENT* consumer only keeps undelivered requests in memory. If writing to the file fails, the batch stays in the queue (`RETRY`); a single request of 24 MB or more is `DISCARDED`. > We recommend using a [cron job](/v1/docs/nodejs-cron-job) to send the requests to the server every few minutes and empty your file to keep it at a manageable size How the FILE consumer works: - Requests are appended to a temporary file `<filePath>/<filePrefix>-<timestamp>.<random>.tmp`. The random part keeps the names unique, so a file created in the same millisecond in which another one was rotated cannot overwrite its log file later. `filePath` is a directory, with or without a trailing path separator. A relative `filePath` is resolved against the current working directory of the process: use an absolute path, especially when the cron job runs from a crontab with another working directory. - Before a batch is written, the file is checked against `maxFileLines`, `maxFileDuration` (in milliseconds) and `maxFileSize` (in bytes). Then the file is renamed to `.log` and a new `.tmp` file is started. If the rename fails, `Create new file, because cannot rename temporary file` is logged at error level and a new `.tmp` file is started anyway. - An existing `<filePrefix>-*.tmp` file in the file path is continued (its line count is read from the file, its age from the timestamp in the file name). A file that was found is opened without creating it; a file that cannot be opened is skipped. Before every write the consumer checks that the open file is still the `.tmp` file (by its inode); if it was renamed or deleted from outside (for example by the cron job), it continues with another existing `.tmp` file or creates a new one. - For the FILE consumer the library always uses a maximum batch size of 1 (every request is written as its own batch); an explicitly configured value other than 1 and the default 50 is logged at level DEBUG. `setMaxBatchSize()` has no effect with the FILE consumer. - All consumers of one process with the same `filePath` and `filePrefix` share **one** temporary file and one line counter, and their writes run one after another. So one tracking or hybrid instance per request is safe. The file stays open for 5 seconds after its last consumer was closed, so the next request does not have to search and open it again; the open file does not keep the process alive. - Several processes (for example cluster or PM2 workers) and the cron job are coordinated with an exclusive lock: the directory `<filePath>/<filePrefix>.lock`, which is created for every write, rotation and takeover by the cron job. A lock that is older than 30 seconds belongs to a process that ended while holding it and is removed. A rename never replaces an existing file, so no request is overwritten or lost. - If writing fails or no file could be opened, the result is `RETRY`: the batch stays in the queue instead of being lost. A single request of 24 MB or more is `DISCARDED`. - The writers and the cron job have to see the same `filePath`. With systemd's `PrivateTmp` the default path `os.tmpdir()` is a different directory for every service: configure a shared directory in that case. ```javascript const config = new MappIntelligenceConfig()    .setTrackDomain("analytics01.wt-eu02.net")    .setTrackId("111111111111111")    .setConsumerType(MappIntelligenceConsumerType.FILE) .setFilePath("path/to/file") .setFilePrefix("MappIntelligenceRequests") .setMaxAttempt(1)    .setAttemptTimeout(100) .setMaxFileLines(5000) .setMaxFileDuration(30 * 60 * 1000) .setMaxFileSize(12 * 1024 * 1024); ``` **CUSTOM** CUSTOM consumers can be used to create your own consumer in order to send the tracking requests. For more information about CUSTOM consumers, see [Interfaces](/v1/docs/nodejs-interfaces). A consumer passed in via `setConsumer()` is used with every consumer type, not only with `CUSTOM`. With the consumer type `CUSTOM` but without a consumer, every flush logs `Cannot send because no consumer is available or an undefined consumer type was specified.` at level ERROR and the requests stay in the queue (`RETRY`). ```javascript class TestCustomConsumer implements IMappIntelligenceConsumer {    public async sendBatch(batchContent: Array): Promise {        // implement custom consumer logic: true = SENT, false = RETRY        return true;    } } const mappIntelligenceConfig = new MappIntelligenceConfig()    .setTrackDomain("analytics01.wt-eu02.net")    .setTrackId("111111111111111")    .setConsumerType(MappIntelligenceConsumerType.CUSTOM)    .setConsumer(new TestCustomConsumer()); ``` --- ## Examples for custom consumers The following examples show CUSTOM consumers for two typical use cases: buffering the requests in Kafka or Redis before they are sent to the track server, and checking your tracking in unit tests. The examples are not part of the library. Adapt them to your environment. ### Kafka Writes every request as one message into a Kafka topic (client library `kafkajs`). `producer.send()` resolves only when the broker has confirmed every message of the batch. If it rejects, the library logs the error and treats the batch as `RETRY`. ```javascript import {Kafka, Producer} from 'kafkajs'; class KafkaTrackingConsumer implements IMappIntelligenceConsumer {    public constructor(private readonly producer: Producer, private readonly topic: string) {}    public async sendBatch(batchContent: Array): Promise {        await this.producer.send({            topic: this.topic,            messages: batchContent.map((request) => ({value: request}))        });        return true;    } } // one producer per process, disconnect it yourself when the process stops const producer = new Kafka({clientId: 'shop', brokers: ['kafka:9092']}).producer(); await producer.connect(); const mappIntelligenceConfig = new MappIntelligenceConfig()    .setTrackDomain("analytics01.wt-eu02.net")    .setTrackId("111111111111111")    .setConsumerType(MappIntelligenceConsumerType.CUSTOM)    .setConsumer(new KafkaTrackingConsumer(producer, 'mapp-intelligence-requests')); ``` ### Redis Appends the requests to a Redis list (client library `ioredis`). `RPUSH` with all requests of the batch is one command, so the batch is appended completely or not at all. If it rejects, the library logs the error and treats the batch as `RETRY`. ```javascript import {Redis} from 'ioredis'; class RedisTrackingConsumer implements IMappIntelligenceConsumer {    public constructor(private readonly redis: Redis, private readonly key: string) {}    public async sendBatch(batchContent: Array): Promise {        await this.redis.rpush(this.key, ...batchContent);        return true;    } } // one client per process, quit it yourself when the process stops; // maxRetriesPerRequest: 1 rejects quickly while Redis is not reachable, instead of keeping the flush waiting const redis = new Redis({host: 'redis', port: 6379, maxRetriesPerRequest: 1}); const mappIntelligenceConfig = new MappIntelligenceConfig()    .setTrackDomain("analytics01.wt-eu02.net")    .setTrackId("111111111111111")    .setConsumerType(MappIntelligenceConsumerType.CUSTOM)    .setConsumer(new RedisTrackingConsumer(redis, 'mapp-intelligence-requests')); ``` ### Forwarding the requests from Kafka or Redis Kafka and Redis only buffer the requests. A separate process of your own (a worker) has to read them and send them to the track server, the same way the HTTP_CLIENT consumer does: - `POST` to `https://<trackDomain>/<trackId>/batch` with the content type `text/plain`, one request per line in the body. - At most 10,000 requests and less than 24 MB per batch. - Handle the response like the HTTP_CLIENT consumer (see the table of batch results above): a status below 400 means delivered; 5xx, 408, 429 and network errors are retried; any other 4xx can never be delivered. - Commit the Kafka offset, or remove the requests from the Redis list, only after the batch was delivered. With Redis and one worker per list, for example read with `LRANGE <key> 0 9999` and after delivery remove the batch with `LTRIM <key> <number of requests> -1`. > [!NOTE] > Note > > If a batch is retried after part of it already reached Kafka or Redis (for example a timeout after the broker has stored the message), these requests are buffered twice. Kafka and Redis do not prevent this, and neither does the worker. If duplicates matter, deduplicate in the worker. ### Test consumer Keeps the requests in memory instead of sending them, so you can check in your own unit tests what the library would track, without network or files. `available = false` simulates an unreachable track server: every batch is then `RETRY` and stays in the queue. ```javascript class InMemoryTestConsumer implements IMappIntelligenceConsumer {    public readonly requests: Array = [];    public available: boolean = true;    public async sendBatch(batchContent: Array): Promise {        if (!this.available) {            return false;        }        this.requests.push(...batchContent);        return true;    } } ``` Usage in a Jest test. `close()` sends the remaining requests, so they are in the consumer once it has resolved: ```javascript test('tracks the page', async () => {    const consumer = new InMemoryTestConsumer();    const mappIntelligenceConfig = new MappIntelligenceConfig()        .setTrackDomain("analytics01.wt-eu02.net")        .setTrackId("111111111111111")        .setConsumerType(MappIntelligenceConsumerType.CUSTOM)        .setConsumer(consumer);    const mappIntelligence = new MappIntelligenceTracking(mappIntelligenceConfig);    await mappIntelligence.track(new MappIntelligenceDataMap().page(new MappIntelligencePage('en.home')));    await mappIntelligence.close();    expect(consumer.requests).toHaveLength(1);    expect(consumer.requests[0]).toContain('en.home'); }); ``` Please check [Configuration](https://docs.mapp.com/v1/docs/nodejs-integration-and-configuration) to see all configuration options and explanations of the Mapp Intelligence Node.js library, including default settings.