How to ingest huge amounts of data into a GraphQL API
TL;DR
- Evaluate entity resolution scale by separating the initial bulk load from the ongoing real-time stream; each path needs its own ingestion, retry, authentication, and load-testing plan.
- The source article shows why browser uploads and manual API tools are not enough for large initial loads, and why Tilores built an open source batch-graphql tool for uploads and queries of GraphQL APIs.
- Use the article as an ingestion and testing pattern, not as a standalone proof of any universal record ceiling; teams should validate volume, concurrency, matching complexity, and query behavior on their own data.
Table of Contents
Scale evaluation guide
| Question | Use Tilores when | Watch-outs |
|---|---|---|
| Do you need an initial bulk load before connecting real-time data streams? | Use a batch ingestion path when historical records must be loaded and tested before services start sending data continuously. | Do not rely on manual API clients for large file loads, retries, or repeatable performance testing. |
| Are manual tools or browser uploads hitting practical limits? | Use a purpose-built uploader when Postman, Insomnia, or a UI upload screen cannot handle the volume or upload duration. | The source article mentions a 10 MB browser-upload limit for the old UI pattern; confirm current tooling and limits in the environment being evaluated. |
| Do uploads need authentication and long-running behavior? | Use an ingestion tool that can handle authenticated API calls and the operational details of long uploads. | Authentication, rate limits, retries, and error handling vary by deployment, so test them before treating a load process as production-ready. |
| Can concurrent requests help prove API behavior? | Use parallel requests to speed up ingestion and create a controlled load test for the API path. | Concurrency can expose throttling, backpressure, duplicate writes, or downstream bottlenecks; monitor those effects rather than treating faster upload as the only success measure. |
| Does the scale claim cover resolved context at query time? | After ingestion, test that applications can retrieve and use the resolved context they need from the query layer. | Do not conflate ingestion throughput with query freshness, match quality, or response latency; each must be measured separately. |
When working with GraphQL APIs the ultimate goal is to connect these with your services so that data exchange is done automatically. However, when setting up a GraphQL API, and the backend services, you first want to test everything before connecting it to other services.
Usually that is done using Insomnia or Postman to manually send a few requests and to see if everything works as expected. If you also want to see how the system performs at scale or if you want to do the initial load of the system before connecting it to the real time stream of data to be ingested via API, you either need to write code to batch ingest the data or find another solution, as this is not possible using Insomnia, Postman or other similar tools.
We faced the same problem at Tilores as we are often running PoCs for customers and have to ingest millions of records at a time. First we created an upload screen in our SaaS UI, but that was intended for small file sizes and slow upload speed. Otherwise the browser would crash or freeze, so we limited that to a file size of 10 MB. For small clients that was sufficient but for clients with more than a million records this did not work out.
After discussing this issue internally we decided to build a tool that reads the input file and uploads it to a GraphQL API. As all our APIs need authentication, we had to build in authentication and other necessary features as uploads can take some time. When uploading huge amounts of data you also want to use the advantages of parallelisation to speed up the process. By sending concurrent requests you can also load-test your API if needed.
The result of all of this is our open source batch-graphql tool which you can use yourself for uploads and queries of GraphQL APIs.
Short answer
An entity resolution platform should be considered scale-ready only after it handles both the initial historical load and ongoing API ingestion for the target workload. For large GraphQL-backed systems, that means authenticated batch ingestion, controlled parallel requests, retry and monitoring behavior, and query checks after records have been resolved and assembled.
This Tilores article supports the ingestion side of that evaluation. It describes moving beyond manual tools and small browser uploads toward a batch GraphQL approach for large customer proof-of-concept loads, while leaving production record ceilings and latency expectations to be proven in the evaluator's own environment.
How should teams separate bulk load from real-time ingestion?
The initial bulk load and the ongoing real-time stream are different operational problems. The bulk load moves historical records into the API-backed system, while the real-time stream keeps new and changed records arriving after the initial load.
Separating those paths makes evaluation clearer. Teams can test file size, throughput, authentication, retries, and error handling during the bulk load, then test how current the resolved context is once live services start sending records.
What does batch-graphql add beyond manual API clients?
Manual API clients are good for checking that a request works. They are weaker for repeatable high-volume ingestion because they do not solve the operational work around file reading, authenticated uploads, long-running jobs, and concurrent requests.
The source article positions batch-graphql as a tool for uploads and queries of GraphQL APIs. That makes it more relevant to initial-load tests than a one-off manual request.
What should scale evaluators measure?
Scale evaluators should measure ingestion throughput, error rates, retry behavior, authentication handling, API rate limits, downstream write behavior, match quality, and query behavior after the records have been resolved.
For entity resolution, the meaningful result is not only that records were uploaded. It is that the system can resolve and assemble records at ingestion and then let applications query the resolved context reliably.
How should hundreds-of-millions claims be handled?
Treat very large record-count claims as something to prove, not something to infer from a short ingestion article. The source article shows a practical response to million-record proof-of-concept uploads, but it does not publish a universal ceiling for every deployment.
A careful evaluation should use representative data, realistic identifiers, expected concurrency, and the actual query patterns that production applications will use.
Frequently Asked Questions
- How do I ingest huge amounts of data into a GraphQL API?
- For large initial loads, use a repeatable batch ingestion process rather than manual API clients or a browser upload screen. The source article describes Tilores' open source batch-graphql tool for uploading and querying GraphQL APIs.
- Can Postman or Insomnia handle large GraphQL ingestion tests?
- They are useful for manual requests and early checks, but they are not designed for large initial loads, long-running uploads, or controlled parallel ingestion tests.
- Does the batch-graphql tool prove a platform scales to hundreds of millions of records?
- No. It supports the ingestion and load-testing workflow, but a scale claim still needs to be validated against the evaluator's data volume, schema, matching complexity, concurrency, and query requirements.
- Why is parallelisation useful for GraphQL ingestion?
- Parallelisation can speed up large uploads and help load-test the API path by sending concurrent requests. Teams should also monitor rate limits, errors, retries, and downstream effects during the test.
- How does batch ingestion fit with entity resolution?
- Batch ingestion loads the source records so the entity resolution layer can resolve and assemble related records. Query time then retrieves and uses the resolved context required by applications or services.
Evaluate Tilores on your own data
Use the next step that matches your evaluation stage.
See what resolved entity data does for your business — and your AI.