create a development environment

[keewis]

• closes Add pytest-accept to a pixi developer environment #10990

We have quite a few environments now, but for development there are a few other tools needed. So far, we have:

• ipython, black, and ipdb
• pytest-accept

Feel free to accept other tools that need access to the environment. (This environment explicitly excludes typing, but it shouldn't be too hard to add another one should we need it).

Note: environments are only materialized once they are accessed, so if you never use the dev env, pixi should never download and install pytest-accept.

cc @Illviljan

[dcherian]

I've started using pdbpp now, and really like it.

Here's a list from my personal dev tool yaml. Can we add

• pdbpp
• line_profiler
• memory_profiler
• memray
• snakeviz
• icecream
• ipykernel
• snoop (haven't tried this one yet, had forgotten about it but it looks great)

Is asv in the dev env?

[keewis]

hmm... looks like it might be better to have a profiling feature / env, in that case. I'd probably include pyinstrument in that list, as well. memory_profiler does not appear to be maintained anymore (not sure if that's a problem).

I don't think asv is anywhere yet, I believe. Is it possible to use that with pixi, or does that have its own dependency installation system? I think I remember it using conda envs before.

[dcherian]

looks like it might be better to have a profiling feature / env

Why not just one big fat one? :P

memory_profiler does not appear to be maintained anymore (not sure if that's a problem).

Interesting. I still find %mprun -f in a notebook is very effective (recent example).

I don't think asv is anywhere yet, I believe. Is it possible to use that with pixi, or does that have its own dependency installation system

I think it still manages its own; but you can tell it to use rattler as a solver. I think it's useful to jsut add asv so we can run benchmarks.

[ruben-arts]

If you see this env as the "default" env for users, you can just rename it to default instead of dev. This would then allow users to run pixi run xxx or pixi shell to start this developer env. And in scripts you can still use -e to specify the more precise env.
If you still want to use the current default env for something else you can rename it with:

new-default = []

Also, you can use solve-group to specify that all these environments should have the same versions for the overlapping dependencies. These guarantees lower file system usage and reproducible environments between the different sets. That would look like this:

typing = { features = ["typing"], solve-group = "main" }
doc = { features = [
  "doc",
  "backends",
  "test",
  "accel",
  "viz",
  "extras",
], solve-group = "main"}
dev = { features = [ "py313",
  "test",
  "backends",
  "accel",
  "numba",
  "dask",
  "viz",
  "extras",
  "dev",
], solve-group = "main"}

If this works, pixi will guarantee that they all use the exact same python and other base dependencies. When they conflict in dependencies, you can still leave out the solve-group.

[VeckoTheGecko]

Thanks for taking the time to look at this @ruben-arts ! (and for responding to my Discord comment about this :)) ) . Also useful for another project I'm working on

[benbovy]

I agree with @Illviljan #10990 (comment) it's nice to have such a "default" development environment with everything + all bells and whistles, especially for onboarding new contributors.

I'm wondering whether it is possible with pixi to define a local-only environment, though. I.e., an environment that is built on top of the Xarray's pixi workspace manifest but that is not included in that manifest (not VCS tracked nor shared with other Xarray contributors). I haven't seen anything in pixi's documentation and repository issues / discussions but I might have missed it?

The motivation is that everyone has its own preferences regarding development tools such as profilers and debuggers. Some of those tools might be already installed globally and could automatically detect virtual environments. Also in a large project like Xarray, contributors (even the regular ones) are rarely working on every feature of the library nor need all the optional dependencies.

Before #10888 I had a local pixi.toml manifest file for Xarray in which I defined a lightweight development environment for my own needs. This was pretty handy, for example I could quickly run all the unit tests locally and skip all tests that are not relevant for the contributions and reviews I usually do (e.g., specific backends, visualization, etc.). Is there a way to keep doing that now that Xarray has an "official" pixi configuration? I guess a workaround would be to continue maintaining my pixi manifest file in a non-default location and use --manifest-path, but that's sub-optimal.

[VeckoTheGecko]

I'm wondering whether it is possible with pixi to define a local-only environment, though ... I guess a workaround would be to continue maintaining my pixi manifest file in a non-default location and use --manifest-path, but that's sub-optimal.

Yes, one way to do that is to use an "out of tree" configuration (this concept is used quite a bit in https://github.com/rgommers/pixi-dev-scipystack/ )

So as opposed to

xarray < -- my workspace folder
├── ...
├── .git
├── pixi.toml
├── pyproject.toml
├── README.md
└── xarray

You can have one level more of nesting

xarray < -- my workspace folder
├── pixi.toml <-- my custom pixi.toml (not in VCS)
└── xarray
    ├── ...
    ├── .git
    ├── pixi.toml
    ├── pyproject.toml
    ├── README.md
    └── xarray

Given Pixi manifest discovery mechanisms - I think its either this, or setting --manifest-path

@lucascolley maybe you want to comment more here

[lucascolley]

I'm wondering whether it is possible with pixi to define a local-only environment, though. I.e., an environment that is built on top of the Xarray's pixi workspace manifest but that is not included in that manifest (not VCS tracked nor shared with other Xarray contributors).

@ruben-arts am I right in thinking that prefix-dev/pixi#4457 is heading towards this use-case?

EDIT: hmm I suppose it addresses the no-lock part, but not the out-of-manifest part

[lucascolley]

I guess a workaround would be to continue maintaining my pixi manifest file in a non-default location and use --manifest-path, but that's sub-optimal.

I think that prefix-dev/pixi#4461 is aimed at making this particular strategy more nice.

[ruben-arts]

@lucascolley I think it would be a combination of the two that is going to help out.

[keewis]

I didn't read the proposals entirely (so they might resolve this need in a different way), but I think it would be useful to have a way to inject globally defined pixi features into locally defined envs. For example, a dev feature containing my preferred profilers / debugger / REPL that would be merged with a local env like the dev env I'm proposing here. Obviously I have no idea how you'd organize the lock file in a way that still can be committed to the repository if desired.

Having a separate workspace (if I'm understanding pixi's definition of workspace correctly) wouldn't work in that case because I'd still want to have the features within the project workspace.

A user-global config similar to this:

[feature.profilers]
...

[feature.debugger]
...

[feature.repl]
ipython = "*"
black = "*"

[environment.extension]
xarray = { path = "/path/to/xarray/repo", name = "dev", features = ["profilers", "debugger", "repl"] }

would already help quite a bit, without turning the dev environment into a kitchen sink (not that sharing the env wouldn't be useful, I've learned about quite a few new tools I had not seen before in this PR).