Document the inexactness of --limit when searching

[dolsysmith]

Running the following command yields 599 Tweets, not 500:

twarc2 search --limit 500 "blacklivesmatter" results.jsonl

Is that expected behavior? (Just wanting to confirm.)

[edsu]

That is the expected behavior yes. It could be documented better though :-)

[igorbrigadir]

Yeah it's because the limit is counted after retrieval, so it pages through results, which by default are 100 max, but can get fewer from the API, so you can get: 99, 100, 100, 100, 100, and then because you're still on 499, which is less than 500, it will have to make 1 extra call and get another 100.