Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new scripts for truncating chunk results #4

Merged
merged 8 commits into from
Apr 4, 2023
Merged

Add new scripts for truncating chunk results #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e59c7d7
New function to truncate dataframes
pedropark99 Apr 4, 2023
3516715
Fixes over the function for printing big dataframes
pedropark99 Apr 4, 2023
787c138
Add script to read and determine all chunk types
pedropark99 Apr 4, 2023
05898b6
Function to truncate big lists is also working
pedropark99 Apr 4, 2023
d24aba5
Function to truncate long texts is also working
pedropark99 Apr 4, 2023
5e43a54
Remove pycache folder
pedropark99 Apr 4, 2023
bec6a34
Remove all unnecessary files
pedropark99 Apr 4, 2023
fb05eb0
Recompile book after last modifications
pedropark99 Apr 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@ metastore_db
Chapters/metastore_db

Chapters/*.html
Chapters/*/*
Chapters/*/*

Scripts/__pycache__/
17 changes: 1 addition & 16 deletions Chapters/04-dataframes.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,29 +63,14 @@ Like any python class, the `DataFrame` class comes with multiple methods that ar

As an example, in the code below I expose all the available methods from this `DataFrame` class. First, I create a Spark DataFrame with `spark.range(5)`, and, store it in the object `df5`. After that, I use the `dir()` function to show all the methods that I can use through this `df5` object:

```{python}
#| include: false
import sys
import os
sys.path.append(os.path.abspath("./../Scripts/"))
from print_big_list import print_big_list
```


```{python}
#| eval: false
#| eval: true
df5 = spark.range(5)
available_methods = dir(df5)
print(available_methods)
```

```{python}
#| echo: false
df5 = spark.range(5)
available_methods = dir(df5)
print_big_list(available_methods)
```


All the methods present in this `DataFrame` class, are commonly referred as the *DataFrame API of Spark*. Remember, this is the most important API of Spark. Because much of your Spark applications will heavily use this API to compose your data transformations and data flows [@chambers2018].

Expand Down
Loading