Better Search Results by kadaan · Pull Request #1871 · jetify-com/devbox

kadaan · 2024-03-01T18:44:46Z

Summary

Output tabular search results which show the platforms available

❯ devbox search ruby
Found 6+ results for "ruby":

┌─────────────┬──────────────────────────────────┬─────────────────┐
│ PACKAGE     │             VERSIONS             │    PLATFORMS    │
├─────────────┼──────────────────────────────────┼─────────────────┤
│             │ 3.3.0                  3.3.0-rc1 │ aarch64-darwin  │
│             │ 3.3.0-preview3    3.3.0-preview2 │ aarch64-linux   │
│ ruby        │ 3.3.0-preview1             3.2.3 │ x86_64-darwin   │
│             │ 3.2.2                      3.2.1 │ x86_64-linux    │
│             │ 3.1.4                      3.1.3 │                 │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┼─────────────────┤
│             │ 0.10.0                           │ aarch64-darwin  │
│ rubyfmt     │                                  │ aarch64-linux   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 0.8.1                            │ aarch64-darwin  │
│             │                                  │ aarch64-linux   │
│             │                                  │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
├─────────────┼──────────────────────────────────┤                 │
│             │ 0.14.0                    0.13.4 │                 │
│             │ 0.13.2                    0.13.0 │                 │
│ ruby-lsp    │ 0.11.1                     0.9.4 │                 │
│             │ 0.8.0                      0.7.4 │                 │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┤                 │
│             │ 5.3.0                            │                 │
│ ruby-zoom   │                                  │                 │
│             │                                  │                 │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┼─────────────────┤
│             │ 0.8.0rc3                         │ aarch64-linux   │
│ rubyripper  │ 0.6.2                            │ x86_64-linux    │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┼─────────────────┤
│ rubyMinimal │ 2.6.6                            │ x86_64-linux    │
│             │                                  │                 │
└─────────────┴──────────────────────────────────┴─────────────────┘

Warning: Showing top 10 results and truncated versions. Use --show-all to show all.

Info: For more information go to: https://www.nixhub.io/search?q=ruby

❯ devbox search ruby --show-all
Found 6+ results for "ruby":

┌─────────────┬──────────────────────────────────┬─────────────────┐
│ PACKAGE     │             VERSIONS             │    PLATFORMS    │
├─────────────┼──────────────────────────────────┼─────────────────┤
│             │ 3.3.0                  3.3.0-rc1 │ aarch64-darwin  │
│             │ 3.3.0-preview3    3.3.0-preview2 │ aarch64-linux   │
│             │ 3.3.0-preview1             3.2.3 │ x86_64-darwin   │
│             │ 3.2.2                      3.2.1 │ x86_64-linux    │
│ ruby        │ 3.1.4                      3.1.3 │                 │
│             │ 3.1.2                      3.1.1 │                 │
│             │ 3.1.0                      3.0.6 │                 │
│             │ 3.0.5                      3.0.4 │                 │
│             │ 3.0.3                            │                 │
│             │                                  │                 │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 3.0.2                            │ aarch64-darwin  │
│             │                                  │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 3.0.1                            │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 3.0.0                            │ x86_64-linux    │
│             │                                  │                 │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.7.8                      2.7.7 │ aarch64-darwin  │
│             │ 2.7.6                      2.7.5 │ aarch64-linux   │
│             │                                  │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.7.4                            │ aarch64-darwin  │
│             │                                  │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.7.3                            │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.7.2                            │ x86_64-linux    │
│             │ 2.7.1                            │                 │
│             │                                  │                 │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.6.8                            │ aarch64-darwin  │
│             │                                  │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.6.7                            │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 2.6.6                            │ x86_64-linux    │
│             │ 2.5.8                            │                 │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┼─────────────────┤
│             │ 0.10.0                           │ aarch64-darwin  │
│ rubyfmt     │                                  │ aarch64-linux   │
│             │                                  │ x86_64-linux    │
│             ├──────────────────────────────────┼─────────────────┤
│             │ 0.8.1                            │ aarch64-darwin  │
│             │                                  │ aarch64-linux   │
│             │                                  │ x86_64-darwin   │
│             │                                  │ x86_64-linux    │
├─────────────┼──────────────────────────────────┤                 │
│             │ 0.14.0                    0.13.4 │                 │
│             │ 0.13.2                    0.13.0 │                 │
│ ruby-lsp    │ 0.11.1                     0.9.4 │                 │
│             │ 0.8.0                      0.7.4 │                 │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┤                 │
│             │ 5.3.0                            │                 │
│ ruby-zoom   │                                  │                 │
│             │                                  │                 │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┼─────────────────┤
│             │ 0.8.0rc3                         │ aarch64-linux   │
│ rubyripper  │ 0.6.2                            │ x86_64-linux    │
│             │                                  │                 │
├─────────────┼──────────────────────────────────┼─────────────────┤
│ rubyMinimal │ 2.6.6                            │ x86_64-linux    │
│             │                                  │                 │
└─────────────┴──────────────────────────────────┴─────────────────┘
Info: For more information go to: https://www.nixhub.io/search?q=ruby

How was it tested?

Ran search locally and looked at the output.

kadaan · 2024-03-05T16:37:46Z

@Lagoja Can someone look at this? I believe that this is a much better search experience for users. Any feedback would be appreciated.

kadaan · 2024-03-05T16:40:09Z

@Lagoja Also, I don't recall the package at the moment, but the current code (and this PR) filter out packages with empty versions. I think that might be wrong, as I think empty version implies "latest" and by filtering them out they are unfindable.

gcurtis · 2024-03-11T18:47:14Z

We appreciate the PR, but I think this change may require a bit more discussion because it has a large impact on the UI.

We also import a couple TUI-like helper libraries already (some indirectly like charmbracelet), so it would be nice to consolidate instead of adding another dependency.

kadaan · 2024-03-11T19:08:14Z

@gcurtis It would be great if some of that discussion could occur here. I'm game to make changes to get a tweaked version of this PR in, as I believe it greatly enhances the usability to the search command.

Also, maybe I missed it, but looks like only lipgloss is currently used from charmbracelet.

mikeland73 · 2024-03-11T19:14:51Z

Yeah, we actually don't pull in any core charmbracelet libraries because they are very large. They would probably increase binary released size by 10MB+

kadaan · 2024-03-11T19:18:50Z

That makes sense. For reference, adding the dependency used here only increased the binary size for Darwin arm64 by 164kb.

Lagoja · 2024-03-11T19:50:06Z

Looking at this, my thoughts are:

I like the fact that this adds some grouping (I thought it was by major/minor versions, but I guess it's by platform). It’s an improvement in readability over our current UI
The table output makes the content a lot longer — it’s a lot more scrolling for search terms with a lot of results, and it may not scan well on smaller terminal windows. I'm going to try it out a bit to see how it works in practice

As an example, if I run devbox search python --show-all, I have to scroll up a lot before I see actual python results. It's a little better if I paginate the output by piping to less

Is there a way to achieve the version grouping while maintaining the compactness of the current output?

kadaan · 2024-03-11T20:00:06Z

@Lagoja Thanks for the feedback. The width is intentionally constrained at the moment. The platforms list could be two columns to reduce the height somewhat. The table dividers, or something like that, seem necessary to be able to unambiguously identify which versions and platforms apply together. Color would be an option, but then it might not work in different terminals or with color blindness.

Another improvement would be to detect the width of the terminal and lay the table out at the width. That would allow more columns for version and platform.

Also, once a package stops flip-flopping between supported platforms the version will just appear in the head row, so that saves a ton of space.

Lagoja · 2024-03-11T20:06:28Z

Would it make sense to have the output paginate by default? That way the most relevant hit is shown first, instead of having to scroll up.

kadaan · 2024-03-11T20:15:53Z

It could. Normal Unix would be to either pipe to a pager or use $PAGER to do the paging.

Lagoja

Ok, my proposal is:

Let's implement pagination. I think it's important to have the most relevant result in the search show up first.
@mikeland73 let's put this behind a feature flag, and advertise it in the upcoming pre-release. If we test it and like it, we can promote it to a default feature.

Thoughts?

kadaan · 2024-04-02T05:05:48Z

I'm game to take a stab at implementing paging if you are all in agreement that it will be a useful addition.

kadaan · 2024-04-02T06:14:22Z

@Lagoja I was considering relying on the system pager ($PAGER) or an devbox specific config ($DEVBOX_PAGER). The could would look like:

	if command, args, ok := findPager(); ok {
		cmd := exec.Command(command, args...)
		cmd.Stdout = w
		pipe, err := cmd.StdinPipe()
		if err != nil {
			return err
		}
		err = cmd.Start()
		if err != nil {
			return errors.WithStack(err)
		}
		defer func(cmd *exec.Cmd) {
			if err := cmd.Wait(); err != nil {
				ux.Ferror(w, "Error running pager: %s\n", err)
			}
		}(cmd)

		w = pipe
		defer func(pipe io.WriteCloser) {
			_ = pipe.Close()
		}(pipe)
	}

kadaan · 2024-05-29T20:59:06Z

This no longer works because v1/search endpoint stopped returning all platforms. To get it to work you can do:

type SearchResultsV2 struct {
	Query        string           `json:"query"`
	TotalResults int              `json:"total_results"`
	Results      []SearchResultV2 `json:"results"`
}

type SearchResultV2 struct {
	Name        string    `json:"name"`
	Summary     string    `json:"summary"`
	LastUpdated time.Time `json:"last_updated"`
}

type PackageV2 struct {
	Name        string    `json:"name"`
	Summary     string    `json:"summary"`
	HomepageURL string    `json:"homepage_url"`
	License     string    `json:"license"`
	Releases    []Release `json:"releases"`
}

type Release struct {
	Version     string     `json:"version"`
	LastUpdated time.Time  `json:"last_updated"`
	Platforms   []Platform `json:"platforms"`
}

type Platform struct {
	Arch          string    `json:"arch"`
	OS            string    `json:"os"`
	AttributePath string    `json:"attribute_path"`
	CommitHash    string    `json:"commit_hash"`
	Date          time.Time `json:"date"`
	Outputs       []Output  `json:"outputs"`
}

type Output struct {
	Name    string `json:"name"`
	Path    string `json:"path"`
	Default bool   `json:"default"`
	Nar     string `json:"nar"`
}

func (c *client) Search(query string) (*SearchResults, error) {
	if query == "" {
		return nil, fmt.Errorf("query should not be empty")
	}

	endpoint, err := url.JoinPath(c.host, "v2/search")
	if err != nil {
		return nil, errors.WithStack(err)
	}
	searchURL := endpoint + "?q=" + url.QueryEscape(query)

	searchResults, err := execGet[SearchResultsV2](context.TODO(), searchURL)
	if err != nil {
		return nil, err
	}

	if searchResults == nil || searchResults.TotalResults == 0 {
		return nil, ErrNotFound
	}

	results := &SearchResults{}
	for _, result := range searchResults.Results {
		endpoint, err = url.JoinPath(c.host, "v2/pkg")
		if err != nil {
			return nil, errors.WithStack(err)
		}
		resolveURL := endpoint + "?name=" + url.QueryEscape(result.Name)

		packageInfo, resolveErr := execGet[PackageV2](context.TODO(), resolveURL)
		if resolveErr != nil {
			return nil, resolveErr
		}

		if packageInfo == nil {
			return nil, ErrNotFound
		}

		results.NumResults++
		versions := make([]PackageVersion, len(packageInfo.Releases))
		for i, release := range packageInfo.Releases {
			systems := make(map[string]PackageInfo, len(release.Platforms))
			for _, platform := range release.Platforms {
				arch := strings.ReplaceAll(platform.Arch, "-", "_")
				if arch == "arm64" {
					arch = "aarch64"
				}
				os := cases.Lower(language.English).String(platform.OS)
				if os == "macos" {
					os = "darwin"
				}
				system := arch + "-" + os
				systems[system] = PackageInfo{
					System: system,
				}
			}
			versions[i] = PackageVersion{
				PackageInfo: PackageInfo{
					Version: release.Version,
				},
				Systems: systems,
			}
		}
		results.Packages = append(results.Packages, Package{
			Name:        packageInfo.Name,
			NumVersions: len(packageInfo.Releases),
			Versions:    versions,
		})
	}

	return results, nil
}

kadaan · 2024-09-10T20:24:10Z

Any reason this was closed?

kadaan · 2026-10-30T14:14:54Z

@mikeland73 I've been running with this from 0.14 till 0.19 and it works great. Can we re-open the discussion on this PR.

mikeland73 · 2026-10-30T16:11:08Z

@kadaan yeah, I'm good with reopening this. Looks like the original fork was deleted so you may need to re-do the branch.

kadaan · 2026-10-30T16:24:41Z

Sweet. I will resend it on top on 19. I would love to not maintain a local fork, aside from Apple notarizing.

kadaan · 2026-10-30T16:52:57Z

@mikeland73 I've noticed that nixsearch.com still supports the v1 and v2 endpoints. The current search functionality goes against v1. But, the package resolve is against v2. The v1 endpoint is better suited to search because it returns all that is needed in one go, whereas in v2 you have to make multiple calls. But, the v2 endpoint correctly returns the systems that can be resolved whereas v1 can return unresolvable systems. Also, I noticed that the v2 search doesn't return some versions that can be installed.

I see that you have issues opened for all these, which should likely land before the search switches to v2.

kadaan force-pushed the kadaan/better_search_results branch from f51334a to 291b5a3 Compare March 1, 2024 18:51

Lagoja reviewed Mar 11, 2024

View reviewed changes

kadaan added 4 commits March 15, 2024 09:56

Output tabular search results which show the platforms available

b59b131

Columnize the versions

700c5c9

Format and lint code

28cd930

Reduce nesting

4e73e8d

kadaan force-pushed the kadaan/better_search_results branch from e797404 to 4e73e8d Compare March 15, 2024 16:56

loreto closed this Sep 10, 2024

Conversation

kadaan commented Mar 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How was it tested?

Uh oh!

kadaan commented Mar 5, 2024

Uh oh!

kadaan commented Mar 5, 2024

Uh oh!

gcurtis commented Mar 11, 2024

Uh oh!

kadaan commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mikeland73 commented Mar 11, 2024

Uh oh!

kadaan commented Mar 11, 2024

Uh oh!

Lagoja commented Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kadaan commented Mar 11, 2024

Uh oh!

Lagoja commented Mar 11, 2024

Uh oh!

kadaan commented Mar 11, 2024

Uh oh!

Lagoja left a comment

Choose a reason for hiding this comment

Uh oh!

kadaan commented Apr 2, 2024

Uh oh!

kadaan commented Apr 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kadaan commented May 29, 2024

Uh oh!

kadaan commented Sep 10, 2024

Uh oh!

kadaan commented Oct 30, 2026

Uh oh!

mikeland73 commented Oct 30, 2026

Uh oh!

kadaan commented Oct 30, 2026

Uh oh!

kadaan commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

kadaan commented Mar 1, 2024 •

edited

Loading

kadaan commented Mar 11, 2024 •

edited

Loading

Lagoja commented Mar 11, 2024 •

edited

Loading

kadaan commented Apr 2, 2024 •

edited

Loading

kadaan commented Oct 30, 2026 •

edited

Loading