- Add test to verify handling of PDF that results in ~1 million tokens after text extraction
- Add test to verify handling of PDF that is ~100 MB big (padded with binary data trash)
For both tests, one challenge is that we likely don't want to keep such files in the git tree, so we need a way to generate them on-the-fly.
For both tests, one challenge is that we likely don't want to keep such files in the git tree, so we need a way to generate them on-the-fly.