TypeError: argument of type 'PosixPath' is not iterable

[den-schmidt]

Bug description

Hi everyone,

I am trying to play with the DAgger example from docs/tutorials/2_train_dagger.ipynb but get the following error when executing the line dagger_trainer.train(2000):

Exception has occurred: TypeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
argument of type 'PosixPath' is not iterable
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/fsspec/core.py", line 323, in _un_chain
    if "::" in path
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/fsspec/core.py", line 383, in url_to_fs
    chain = _un_chain(url, kwargs)
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1515, in save_to_disk
    fs, _ = url_to_fs(dataset_path, **(storage_options or {}))
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/imitation/data/serialize.py", line 23, in save
    huggingface_utils.trajectories_to_dataset(trajectories).save_to_disk(p)
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/imitation/algorithms/dagger.py", line 147, in _save_dagger_demo
    serialize.save(npz_path, [trajectory])
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/imitation/algorithms/dagger.py", line 285, in step_wait
    _save_dagger_demo(traj, traj_index, self.save_dir, self.rng)
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 206, in step
    return self.step_wait()
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/imitation/data/rollout.py", line 447, in generate_trajectories
    obs, rews, dones, infos = venv.step(acts)
  File "/home/denni/msc/imitation/.venv/lib/python3.8/site-packages/imitation/algorithms/dagger.py", line 669, in train
    trajectories = rollout.generate_trajectories(
  File "/home/denni/msc/imitation/test.py", line 41, in <module>
    dagger_trainer.train(2000)
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
TypeError: argument of type 'PosixPath' is not iterable

Steps to reproduce

I would be really thankful if someone could try out the steps to see if that is only my issue. To reproduce the bug, please try out the file docs/tutorials/2_train_dagger.ipynb as I did on my personal machine as well as in Google Colab.

Environment

• Personal Machine/Google Colab
  • Operating system and version: WSL with Ubuntu 22.04/Ubuntu 22.04
  • Python version: 3.8.19/3.10
  • Output of pip freeze --all: I just installed imitation + this apt-get install ffmpeg freeglut3-dev xvfb beforehand. /Same for Google Colab

absl-py==2.1.0
aiohttp==3.9.5
aiosignal==1.3.1
alembic==1.13.1
async-timeout==4.0.3
attrs==23.2.0
cachetools==5.3.3
certifi==2024.2.2
charset-normalizer==3.3.2
cloudpickle==3.0.0
colorama==0.4.6
colorlog==6.8.2
contourpy==1.1.1
cycler==0.12.1
datasets==2.19.1
dill==0.3.8
docopt==0.6.2
Farama-Notifications==0.0.4
filelock==3.14.0
fonttools==4.52.4
frozenlist==1.4.1
fsspec==2024.3.1
gitdb==4.0.11
GitPython==3.1.43
google-auth==2.29.0
google-auth-oauthlib==1.0.0
greenlet==3.0.3
grpcio==1.64.0
gymnasium==0.29.1
huggingface-hub==0.23.2
huggingface-sb3==3.0
idna==3.7
imitation==1.0.0
importlib_metadata==7.1.0
importlib_resources==6.4.0
Jinja2==3.1.4
joblib==1.4.2
jsonpickle==3.0.4
kiwisolver==1.4.5
Mako==1.3.5
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.7.5
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
munch==4.0.0
networkx==3.1
numpy==1.24.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.40
nvidia-nvtx-cu12==12.1.105
oauthlib==3.2.2
optuna==3.6.1
packaging==24.0
pandas==2.0.3
pillow==10.3.0
pip==24.0
protobuf==5.27.0
py-cpuinfo==9.0.0
pyarrow==16.1.0
pyarrow-hotfix==0.6
pyasn1==0.6.0
pyasn1_modules==0.4.0
pygame==2.5.2
Pygments==2.18.0
pyparsing==3.1.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.7.1
rsa==4.9
sacred==0.8.5
scikit-learn==1.3.2
scipy==1.10.1
seals==0.2.1
setuptools==56.0.0
six==1.16.0
smmap==5.0.1
SQLAlchemy==2.0.30
stable_baselines3==2.3.2
sympy==1.12.1
tensorboard==2.14.0
tensorboard-data-server==0.7.2
threadpoolctl==3.5.0
torch==2.3.0
tqdm==4.66.4
triton==2.3.0
typing_extensions==4.12.0
tzdata==2024.1
urllib3==2.2.1
wasabi==1.1.2
Werkzeug==3.0.3
wheel==0.43.0
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4
zipp==3.19.0

[eufrizz]

I get the same. Here is the full stack trace:

{
	"name": "TypeError",
	"message": "argument of type 'PosixPath' is not iterable",
	"stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[106], line 11
      2 print(tmpdir)
      3 dagger_trainer = SimpleDAggerTrainer(
      4     venv=env,
      5     scratch_dir=tmpdir,
   (...)
      8     rng=np.random.default_rng(),
      9 )
---> 11 dagger_trainer.train(2000)

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/imitation/algorithms/dagger.py:669, in SimpleDAggerTrainer.train(self, total_timesteps, rollout_round_min_episodes, rollout_round_min_timesteps, bc_train_kwargs)
    662 round_timestep_count = 0
    664 sample_until = rollout.make_sample_until(
    665     min_timesteps=max(rollout_round_min_timesteps, self.batch_size),
    666     min_episodes=rollout_round_min_episodes,
    667 )
--> 669 trajectories = rollout.generate_trajectories(
    670     policy=self.expert_policy,
    671     venv=collector,
    672     sample_until=sample_until,
    673     deterministic_policy=True,
    674     rng=collector.rng,
    675 )
    677 for traj in trajectories:
    678     self._logger.record_mean(
    679         \"dagger/mean_episode_reward\",
    680         np.sum(traj.rews),
    681     )

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/imitation/data/rollout.py:447, in generate_trajectories(policy, venv, sample_until, rng, deterministic_policy)
    444 while np.any(active):
    445     # policy gets unwrapped observations (eg as dict, not dictobs)
    446     acts, state = get_actions(obs, state, dones)
--> 447     obs, rews, dones, infos = venv.step(acts)
    448     assert isinstance(
    449         obs,
    450         (np.ndarray, dict),
    451     ), \"Tuple observations are not supported.\"
    452     wrapped_obs = types.maybe_wrap_in_dictobs(obs)

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py:206, in VecEnv.step(self, actions)
    199 \"\"\"
    200 Step the environments with the given action
    201 
    202 :param actions: the action
    203 :return: observation, reward, done, information
    204 \"\"\"
    205 self.step_async(actions)
--> 206 return self.step_wait()

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/imitation/algorithms/dagger.py:285, in InteractiveTrajectoryCollector.step_wait(self)
    277 fresh_demos = self.traj_accum.add_steps_and_auto_finish(
    278     obs=next_obs,
    279     acts=self._last_user_actions,
   (...)
    282     dones=dones,
    283 )
    284 for traj_index, traj in enumerate(fresh_demos):
--> 285     _save_dagger_demo(traj, traj_index, self.save_dir, self.rng)
    287 return next_obs, rews, dones, infos

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/imitation/algorithms/dagger.py:147, in _save_dagger_demo(trajectory, trajectory_index, save_dir, rng, prefix)
    143 npz_path = save_dir / filename
    144 assert (
    145     not npz_path.exists()
    146 ), \"The following DAgger demonstration path already exists: {0}\".format(npz_path)
--> 147 serialize.save(npz_path, [trajectory])
    148 logging.info(f\"Saved demo at '{npz_path}'\")

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/imitation/data/serialize.py:23, in save(path, trajectories)
     16 \"\"\"Save a sequence of Trajectories to disk using HuggingFace's datasets library.
     17 
     18 Args:
     19     path: Trajectories are saved to this path.
     20     trajectories: The trajectories to save.
     21 \"\"\"
     22 p = util.parse_path(path)
---> 23 huggingface_utils.trajectories_to_dataset(trajectories).save_to_disk(p)
     24 logging.info(f\"Dumped demonstrations to {p}.\")

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/datasets/arrow_dataset.py:1515, in Dataset.save_to_disk(self, dataset_path, fs, max_shard_size, num_shards, num_proc, storage_options)
   1512 num_shards = num_shards if num_shards is not None else num_proc
   1514 fs: fsspec.AbstractFileSystem
-> 1515 fs, _ = url_to_fs(dataset_path, **(storage_options or {}))
   1517 if not is_remote_filesystem(fs):
   1518     parent_cache_files_paths = {
   1519         Path(cache_filename[\"filename\"]).resolve().parent for cache_filename in self.cache_files
   1520     }

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/fsspec/core.py:383, in url_to_fs(url, **kwargs)
    372 known_kwargs = {
    373     \"compression\",
    374     \"encoding\",
   (...)
    380     \"num\",
    381 }
    382 kwargs = {k: v for k, v in kwargs.items() if k not in known_kwargs}
--> 383 chain = _un_chain(url, kwargs)
    384 inkwargs = {}
    385 # Reverse iterate the chain, creating a nested target_* structure

File ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/fsspec/core.py:323, in _un_chain(path, kwargs)
    319 def _un_chain(path, kwargs):
    320     x = re.compile(\".*[^a-z]+.*\")  # test for non protocol-like single word
    321     bits = (
    322         [p if \"://\" in p or x.match(p) else p + \"://\" for p in path.split(\"::\")]
--> 323         if \"::\" in path
    324         else [path]
    325     )
    326     # [[url, protocol, kwargs], ...]
    327     out = []

TypeError: argument of type 'PosixPath' is not iterable"
}

[tomtseng]

Fixed in #831 (comment)