Skip to content

CLI Errors fail to show for invalid input file #685

@igorbrigadir

Description

@igorbrigadir

If an invalid input file like input.txt with quotes is used:

"78021221"
"20452058"
"20434358"

The errors are recorded in twarc.log

2023-01-16 00:02:09,705 WARNING invalid username or user id ""78021221"" on line 1
2023-01-16 00:02:09,705 WARNING invalid username or user id ""20452058"" on line 2
2023-01-16 00:02:09,705 WARNING invalid username or user id ""20434358"" on line 3

But in CLI it looks like it succeeded, which is confusing:

twarc2 timelines --use-search input.txt output.jsonl
100%|█████████████| Processed 3/3 lines of input file [00:00<00:00, 3866.91it/s]

This is also related to #561 - if a line in the input fails, it should show an error, or show a summary at the end showing what failed.

Separately, it may be worth supporting quoted / unquoted input files?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions