fix: data being shipped in `MANIFEST.in`

[cadenmyers13]

Problem

This line ships data with labpdfproc,

include src/diffpy/labpdfproc/data *

Proposed solution

Do we want to move away from this? we should find a solution

[sbillinge]

I think in this case we may not want to move away from it because it is data used by the code such as x-ray absorption tables, right? If that is the case, then we may want to google around for recommended ways of doing this. But if it is data for examples and docs, then we do want to move away from it.

[cadenmyers13]

@sbillinge it looks like its okay to ship these because they are relatively small files and required for the package to function. I will close this

[cadenmyers13]

closing as we are not changing this

[sbillinge]

I wanted to get more clarity on this issue so I had a convo with chatgpt and I reproduce it here. Please could I ask you to revisit this issue here and let's follow the guidelines, and also quickly revisit skpkg and make sure we are making the right recommendations there? I think this convo laid things out pretty clearly for me, especially the comment part way down about which things go in sdist + whl, which things just in sdist, and when to make it a separate download. V. helpful.

Absolutely. Below is the same content cleanly formatted in GitHub-flavored Markdown (GFM), ready to paste directly into a GitHub Discussion. No special extensions required.

---

Best practice for including static data files in a pure-Python package

This summarizes current best practice (modern setuptools, PEP 517/518) for including static data files in a pure-Python, open-source package, with attention to:

1. What should go into the sdist vs wheel

2. How to configure MANIFEST.in

3. How to configure pyproject.toml

---

1. Conceptual best practice

Recommended rule of thumb

If data is required at runtime, include it in both the sdist and the wheel.

Reasons:

• Wheels are the dominant install artifact (pip install almost never builds from sdist)

• Missing data in wheels causes subtle runtime failures

• There is no downside for pure-Python packages

Only include data in the sdist but not the wheel if it is not required at runtime, e.g.:

Data type | Where it belongs -- | -- Runtime configuration / schemas | sdist + wheel Test data | sdist only Examples / benchmarks | sdist only Docs | sdist only Large reference files | sdist or external download

Your statement that sdist-only data is acceptable is correct only if the package does not rely on those files at runtime.

---

2. Mental model: how packaging actually works

There are two independent mechanisms:

MANIFEST.in

Controls what goes into the sdist (.tar.gz)

package-data / include-package-data

Controls what goes into the wheel (.whl)

They serve different purposes and do not substitute for one another.

---

3. Recommended project layout for static data

Place static data inside the package:

src/
  mypkg/
    __init__.py
    core.py
    data/
      defaults.yaml
      schema.json
      lookup.csv

Why this layout:

• Keeps data versioned with code

• Works with importlib.resources

• Works identically from wheel, sdist, editable installs, and zip imports

Avoid top-level data/ directories for runtime files.

---

4. Runtime access (critical)

Do not rely on filesystem paths relative to __file__.

Use importlib.resources:

from importlib.resources import files
data_path = files("mypkg.data") / "defaults.yaml"

text = data_path.read_text()

This works correctly whether:

• installed from wheel

• installed from sdist

• installed editable

• installed as a zip archive

This is now the canonical approach.

---

5. pyproject.toml (modern setuptools setup)

[build-system]

requires = ["setuptools>=67", "wheel"]

build-backend = "setuptools.build_meta"

[project]

name = "mypkg"

version = "0.1.0"

requires-python = ">=3.9"
[tool.setuptools]

package-dir = {"" = "src"}

include-package-data = true
[tool.setuptools.packages.find]

where = ["src"]

Key points:

• include-package-data = true allows MANIFEST.in to influence wheel contents

• src/ layout is explicitly declared

---

6. MANIFEST.in (sdist control)

To include all static data files in the sdist:

recursive-include src/mypkg/data *

More selective version:

recursive-include src/mypkg/data .yaml .json .csv

Common additions:

include LICENSE

include README.md

---

7. Wheel inclusion options

There are two correct approaches. Choose one.

Option A (preferred): let MANIFEST.in drive everything

[tool.setuptools]

include-package-data = true

Pros:

• Single source of truth

• Simple

• Most common modern setup

---

Option B (explicit): declare package data explicitly

[tool.setuptools.package-data]

mypkg = ["data/.yaml", "data/.json", "data/.csv"]

Use this if:

• You want wheels and sdists to differ

• You prefer explicit declarations over implicit behavior

---

8. Intentional sdist-only data

This is appropriate for tests, examples, etc.

MANIFEST.in

recursive-include tests *

recursive-include examples *

Do not list these paths in package-data.

Result:

• Available to source builds and developers

• Not shipped in wheels

---

9. Common mistakes to avoid

• Relying on MANIFEST.in alone

• Using file to locate data

• Putting runtime data outside the package

• Assuming pip installs from sdists

• Using legacy setup.py hacks

---

10. TL;DR

Best practice today:

• Runtime data → include in sdist + wheel

• Use importlib.resources

• Declare data via MANIFEST.in

• Enable include-package-data = true

Minimal setup:

# MANIFEST.in

recursive-include src/mypkg/data *

# pyproject.toml

[tool.setuptools]

include-package-data = true

---

If you want, I can also:

• Review an actual repo layout

• Show python -m build output to verify contents

• Provide a checklist for migrating from setup.cfg or setup.py

---

If you want this slightly shorter or more opinionated for a GH discussion thread (e.g. “accepted answer” style), I can tighten it further.

[cadenmyers13]

If data is required at runtime, include it in both the sdist and the wheel.

@sbillinge from this comment, the current packaging is correct as this is required at runtime.

Do not rely on filesystem paths relative to file.

Addressed this in the attached PR