Arrow X Datasets Blog

[cakiki]

No description provided.

[lhoestq]

Awesome ! Just a few comments on the code blocks formatting and suggestions :)

[lhoestq]

Suggested change

	```python
	python = dset.map(lambda table: table.filter(pc.field("lang") == "Python"), batched=True, batch_size=500_000, num_proc=10)
	print(f'{python.num_rows:,}', "rows")
	>>> 250,000 rows
	CPU times: user 818 ms, sys: 255 ms, total: 1.07 s
	Wall time: 3.46 s
	```
	```python
	>>> %time python = dset.map(lambda table: table.filter(pc.field("lang") == "Python"), batched=True, batch_size=500_000, num_proc=10)
	CPU times: user 818 ms, sys: 255 ms, total: 1.07 s
	Wall time: 3.46 s
	>>> print(f'{python.num_rows:,}', "rows")
	250,000 rows
	```

[mariosasko]

Maybe we can use the standard lib for measuring time instead of the magic commands so that this code works in the (standard) REPL

[mariosasko]

Nice! Some comments: