Feedback request: how can we make pyinfra’s documentation better

[Fizzadar]

I’d like to improve pyinfra’s documentation, I think there’s a few things that open come up and clearly need documenting better. Off the top of my head some of these are:

• operation execution (when do things execute)
• how to get output from an operation
• how to read system data before operations execute (ie the server.Command fact)

Would be amazing to hear what other things are unclear to people, and any ideas on how to better document or structure the docs would be much appreciated! 🙏

[goetzk]

Some docs related comments from me:

Developing and using inventory and execution connectors is not obvious to me ; i've started documentation for that at https://github.com/pyinfra-dev/pyinfra/compare/3.x...medeopolis:pyinfra:3.x-documentation-updates?expand=1 and will share it once i've confirmed its correct.

Outstanding questions which i haven't added to that doc (because i don't have the answers yet)

• if a inventory connector needs configuration (to access an API), how should the configuration be 'inserted'. environment? config file? to be confirmed...
• how to use pyinfra logger appropriately , eg logging at -v , -vv, -vvv (this might be obvious from the code but i haven't checked the code yet)
• how to package functions (connector, fact gathering, operations) for use with pyinfra (including recommended tools, configuration which needs to be created, paths to observe, etc)

Edit: on the subject of developing connectors:
There are lots of helpers/formatters available in pyinfra that are used during connector development. its possible to cargo cult most of the process but it would still be good to have more """python comments""" explaining what things do.

[vram0gh2]

Hi - Firstly, I must thank you and all the contributors to the project. Out of the various configuration automation systems out there, this fills an important role. It's approachable, uses an actual scripting language for specifying what is to be done, only executes shell commands on the remote systems (agentless), and puts idempotency and declarative end state up-front. Thank you!

I just started with pyinfra; for me, below are some missing, confusing, and ambiguous pieces in the docs. If you are familiar with other config automation systems, you figure most of this out, but it shouldn't be expected or required. Warning: this is a long post.

Top-level

The homepage mentions a lot of positive traits/differentiators pyinfra possesses, but to see what that really means, you have to go all over the docs. Two ideas for improvement:

• The top level index section of the docs could have a subsection mimicking the homepage, but link to the places in the docs that talk about those characteristics.
• Use those traits as an explicit checklist of items the docs should discuss, ideally with consistent phrasing as with the homepage.

Getting Started

• Explicitly say whether the naming or quantity of host lists matter. What does pyinfra do in terms of iterating over multiple variables, how are those names used? What happens if a host is in two lists at the same time, does that matter?
• Create a Deploy raises many questions, but then says "That’s the basics of pyinfra!". It feels glib and untrue. You don't have to answer the following right there, but at least link to where these are discussed in the detailed chapters, and be sure to address them in those chapters. Perhaps take the mini-index there and say for each of those index links, where the following are discussed:
  • Do the filenames of the CLI arguments matter or is it purely positional?
  • What are the requirements on the remote system for the command to be executed?
  • What identity is used to login to the system and perform any commands? Will you get prompted for a password on the remote system if you don't use a key-based login? If some commands can only be executed as root but the login identity isn't root, does pyinfra identify this beforehand and do something automagically (e.g., use sudo or doas) or alert the user?
    • Probably a good place for any admonishments about directly logging in as root.
  • The code block has a comment, # Define some state - this operation will do nothing on subsequent runs. Why will it do nothing on subsequent runs? Is there a cache of host state on the system running pyinfra? Maybe change this to something more descriptive, like # Define end state - checked each run, action taken only if not satisfied? Or something else more concise, but comprehensive and clear?

Using Operations

• The "What are operations?" block is confusing/ambiguous. Maybe just explicitly say up-front that operations can either be a) imperative actions to be taken, or b) declaratively identify an end state to be achieved, in which case pyinfra figures out if anything needs to be done to reach that end state, and if so, does it.
• It can be confusing when the name argument in the examples begin with imperative verbs, which may suggest it will attempt to do this each time it's run. But because these operations are idempotent, that's not the case. Say what the name argument is used for?
• Early on, perhaps right after the first example, show how this gets turned into shell commands to be executed on the host?
• Before getting into the The host Object, perhaps discuss how the operations check the current state and only act if needed. This can reinforce that things do get executed on the host to check the state and there can be an example case of what happens when the state matches the operation end state and when it doesn't. In the rest of the page, what it means to "execute an operation" for a declarative operation is unclear. Is it an atomic check of state and immediate action if the target end state is not met? Does it follow a phased approach where first host state is checked, and in a separate phase all actions to be taken are executed? If the latter, does the user know after the change detection and after and after any host state mutation?
• Host Facts: it may help to remind the user that for individual declarative end state operations, the user doesn't have to check the fact associated with the current state themselves to determine if action is required for that operation.
• Host Facts: there is an admonishment to only use immutable facts in deploy code. Spell out what immutable means in this context?
• Fact Errors: it is mentioned "the host will be marked as failed just as it would when an operation fails". But we haven't yet talked about what happens when an operation fails, how would the user already understand that?
• Operation Changes & Output: The first sentence is confusing without understanding pyinfra's execution model. At least link to that section first? When it is said that "all operations return ..." that implies the operation is complete, but then the sentence says "changes the operation will execute." This sounds like an intuitive contradiction as written.
• Is .did_change the only signal field these operations return? It's just used in the example without discussing it first.
• The config Object: When should the config object be modified vs. using global variables?
• When examining the host state to see if modifications are needed, what happens if the desired end state is the compound effect of multiple operations? I am thinking about files for example, where you might upload a file and then modify it in a subsequent operation.

Inventory & Data

• Data Hierarchy: mentions all.py - where is that discussed before? Is this a special file that is read outside of the two arguments provided to the CLI?

Global Arguments

• Operation meta & callbacks: missed opportunity to discuss how the .name field is used.
• Execution strategy: Which is the default strategy, what does a value of 0 for ._parallel mean?

FAQ

• How do I chmod or chown ...: is that a stray all caps "LINK" in the answer?

Deploy Reference -> Operations Index

• The opening sentence sounds imperative in nature, rather than clearly either imperative or declarative. It says it "describe[s] changes to make to systems" but if the system already meets the specified target state, no change is made. This is the same point noted above regarding the "Using Operations" section. It is modified by the second sentence, but overall it could be clearer. Maybe change the opening sentence to something like "Operations execute specified commands or ensure specified target state conditions on the systems in the inventory."
• Files Operations: these are probably feature requests masquerading as doc comments, but it's surprising there is no file operation to apply patches, or set mtime values. I see Add files.patch operation #877, but nothing obvious on setting mtime. Are these "patches welcome" areas, or are they specifically being excluded?

How pyinfra works

• In general, this whole section doesn't seem user-focused. Subsection titles seem like misnomers, and the examples don't comprehensively illustrate what is being said. It is hard to build a mental model of what is really going on, and when it is happening (sequencing/phasing), on both the pyinfra side and remote host side based on what is here.
• How pyinfra Detects Changes & Orders Operations
  • Does this section actually explain what the section title says it does? What I get from it is that execution of operations is broken down into a series of parallel execution blocks, but no clear understanding of "why the prepare / ordering stage is so critical" beyond that. There are no specifics as to the change detection code running on the host itself. Maybe a more apt title would be along the lines of "How pyinfra builds a sequence of parallel [operations] execution blocks"?
  • It's not clear as-written why "if we do it in parallel we can’t ensure operations run in expected order" matters. On a per-host basis, if you built a host-specific chain of all the operations, you could conceivably execute those per-host chains in parallel. I believe what is meant here is that if operations on a given host are dependent on operations that would have executed on another host previously, based on how it's sequenced in the deploy, the intended sequence could be violated by running all these execution chains in parallel. This could really benefit from a more comprehensive example and/or diagram right there. It seems the When does this matter? tries to explain this, but deferring the explanation and using a non-comprehensive example instead just adds to the murkiness.
  • The penultimate sentence in the section is also confusing when it says "we have to first execute the code to generate the order" - is there any part of the deploy itself that actually gets executed beyond developing the host block sequence for execution? This confusion is compounded when it says "This is why deploy code is always executed before any changes are made." I think what this is saying is that the act of running the deploy code creates the sequence of parallel operation execution blocks. The way this occurs is that invoking an operation doen't immediately execute any actions on the host (other than maybe gathering facts?). Instead, the operations return deferred execution handles. After the whole block sequence is constructed, the sequence is walked by pyinfra and the handles are invoked to execute actions. Is that essentially correct? If so, I think it is less confusing if explained that way.
• If the above is essentially correct, then how are the state checking vs. host modification steps layered in? Is the execution of an operation "atomic", in that the state condition is checked and the host immediately modified if the state isn't the target, or is all state checking done in a batch per block, then modifications done in a second phase of that batch? Something else?
• There is a missed opportunity here to discuss how operations are turned into host shell code.
• Is a host only accessed once (e.g., single ssh session) over the course of a whole deploy, or are there circumstances under which a sequence of connections to the host are made?

If you read this far, thank you. At least for me, if the above were addressed, I'd be a lot clearer on how pyinfra works and how best to use/not use it.