Can We Measure the Quality of a Curated Resource?

In my previous article, I argued that Awesome Lists need more than links.

They need curation, structure, governance, maintenance and clear standards for deciding what belongs in a collection.

That argument leads naturally to a harder question:

If we want to prioritize quality over quantity, how do we actually evaluate quality?

It is easy to say that a curated collection should contain high-quality resources. It is much harder to define what “high quality” means consistently enough to use it when reviewing real submissions.

A project might be extremely relevant but poorly documented. Another might have excellent documentation but show little evidence of ongoing maintenance. A commercial tool might be polished and widely used but submitted with language that reads more like advertising than a neutral resource description.

Then there are projects that simply fall somewhere in the middle.

They are not obviously good enough to accept but they are not obviously bad enough to reject.

Maintainers make these decisions all the time.

The challenge is that the criteria behind those decisions often remain informal. We develop instincts. We recognize patterns. We remember what we accepted previously. We compare a submission against what is already in the collection.

That human judgment is valuable and I do not think it should disappear.

But I have been exploring whether we can give that judgment more structure.

That is the idea behind ResourceRank.

Imagine maintaining a curated collection that receives regular submissions.

One contributor submits an established open source project with extensive documentation, recent releases, a clear licence, examples, tests and an obvious connection to the collection.

Another submits a newly launched commercial product. It fits the topic but there is limited evidence of adoption, the description is highly promotional and the contributor is affiliated with the project.

A third submission is an excellent resource that has not been updated recently.

A fourth is popular but only loosely related to the category where the contributor wants it listed.

How should these resources be evaluated?

Most maintainers can probably form an opinion about each one relatively quickly.

But explaining exactly why one should be accepted and another should receive additional review is more difficult.

The problem becomes more significant as a collection grows, as multiple maintainers become involved or as decisions accumulate over several years.

Without a framework, two very similar resources can receive different treatment because they were reviewed at different times, by different people, or under different assumptions.

That is not necessarily evidence of bad maintainership. Curation is inherently contextual.

But greater consistency is possible.

One tempting solution is to rely on easily measurable indicators.

For an open source project, that might mean GitHub stars, forks, contributors, releases or activity.

For a website, it might mean traffic or domain authority.

For a newsletter, it might mean subscriber count.

These signals can tell us something.

They cannot tell us everything.

A project with 20,000 GitHub stars is not automatically more useful than one with 500. A new project may be excellent before it develops significant adoption. A specialized resource may serve a small community extremely well. A widely shared resource may have poor documentation or no longer be maintained.

Popularity can also reinforce existing visibility.

Resources that are already well known accumulate more attention, which creates stronger popularity signals, which can then be interpreted as evidence that they deserve even more attention.

That can make discovery systems increasingly conservative.

For curation, I think popularity should therefore be treated as one signal among several, not as a substitute for evaluation.

The same applies to any individual metric.

There is no single number that tells us whether a resource deserves someone's attention.

Before building ResourceRank, I needed to think about the characteristics I was already considering when reviewing resources.

I eventually organized them into six broad signals:

None of these signals provides the answer by itself.

Together, however, they create a more useful picture.

Consider an open source tool that is highly relevant, well documented, actively maintained, appropriately licensed and neutrally presented but relatively new.

Its trust or adoption indicators might still be limited.

That should not necessarily result in rejection.

Instead, the evaluation can surface that limitation alongside the project's strengths and leave the final decision to the maintainer.

That distinction became important in designing ResourceRank.

I did not want to create a system that pretends curation can be automated into a simple yes-or-no calculation.

I wanted a system that makes the factors behind a decision more visible.

ResourceRank is a structured review tool for evaluating whether a submitted resource belongs in a curated collection.

At its simplest, the maintainer provides structured information about a resource and chooses a review profile.

ResourceRank evaluates the resource against the profile and produces a structured review.

That review includes:

The score is useful but it is deliberately not the entire output.

In fact, I think the information surrounding the score can be more useful than the score itself.

If a resource receives 61 out of 100, the important question is not simply whether 61 is “good.”

The useful questions are:

That turns a number into a review process.

ResourceRank currently uses three broad outcomes.

An accept result means the resource appears strong enough for inclusion under the selected review profile, assuming the maintainer agrees with the proposed placement and description.

It is not an automatic approval.

The maintainer still owns the decision.

I think review may actually be the most important outcome in the system.

A resource in this range shows enough value to warrant consideration but one or more issues deserve human attention.

Perhaps the project is highly relevant but new.

Perhaps documentation is incomplete.

Perhaps licensing information is unclear.

Perhaps the resource was submitted by its creator and the presentation is unusually promotional.

Perhaps the available evidence is simply insufficient to make a confident determination.

Instead of forcing the system to classify the resource as good or bad, review acknowledges uncertainty.

That is much closer to how real curation works.

A reject result indicates that the resource does not currently meet the configured threshold or has concerns that make inclusion inappropriate under that profile.

Again, that does not mean ResourceRank has veto power.

It means the structured evidence suggests that the maintainer should have a strong reason for overriding the result.

One feature I wanted ResourceRank to include from the beginning was confidence.

Scoring systems can create a false sense of precision.

A score of 72 can look authoritative even when it was calculated from incomplete information.

ResourceRank therefore considers how much information was available when producing its assessment.

That distinction matters.

A resource scoring 75 with extensive information about its documentation, maintenance, licensing, releases, examples, tests and other characteristics is different from a resource scoring 75 when half of those fields are unknown.

The score may be identical.

Our confidence in what that score represents should not be.

This is one way of acknowledging an important limitation of structured evaluation: missing information is itself information.

Sometimes the correct outcome is not “this resource is weak.”

It is “we do not yet know enough about this resource.”

Quality is also contextual.

A resource can be excellent and still not belong in a particular collection.

This is why ResourceRank includes category fit rather than treating quality as an abstract characteristic of the resource.

Imagine an exceptional data visualization library submitted to an Awesome List focused specifically on database administration.

The library could be mature, well documented, actively maintained, trusted and open source.

It still might not belong there.

This sounds obvious but it illustrates something important about curation.

We are not really asking:

“Is this a good resource?”

We are asking:

“Is this a good resource for this collection?”

And often:

“Is this a good resource for this particular section of this collection?”

Context changes the evaluation.

That is one reason I do not think a universal quality score would be particularly useful.

ResourceRank addresses this through review profiles.

A profile defines how much weight different signals should receive and where the thresholds for accept, review and reject should sit.

The project currently includes profiles for:

This is where ResourceRank started becoming more interesting to me than an Awesome List utility.

The underlying problem exists across many forms of curation.

A newsletter editor deciding whether a source belongs in a recurring research workflow is making a curation decision.

Someone maintaining a public dataset catalog is making curation decisions.

A learning library, research repository, software directory, knowledge base or resource collection faces similar questions.

But the criteria should not necessarily be weighted equally.

For an Awesome List, relevance and documentation may receive significant weight.

For a dataset catalog, openness may deserve greater importance.

For a newsletter source, licensing may be much less important than relevance, trust and quality.

The resource itself has not changed.

The curation context has.

Profiles allow the evaluation model to reflect that.

One of the principles I want to preserve as ResourceRank develops is explainability.

If a system recommends rejecting a resource, the maintainer should be able to understand why.

That is why the current version is intentionally rules-based.

ResourceRank v1 does not crawl GitHub automatically. It does not inspect websites autonomously. It does not send submissions to a large language model and ask whether they look good.

The input is structured. The weights are visible. The thresholds are visible. The resulting signals can be inspected.

That makes the system less sophisticated in some ways, but considerably easier to understand.

For an early version, I think that is the correct trade-off.

A curation system needs consistency before complexity.

If the basic schema, review criteria, scoring philosophy, and output format do not make sense, adding AI will not fix the underlying problem.

It will simply make the decision process harder to inspect.

This point is important enough to state plainly:

ResourceRank does not decide what belongs in a collection.

The maintainer does.

ResourceRank is decision support.

It can identify signals.

It can apply a consistent weighting model.

It can flag concerns.

It can make comparisons easier.

It can create a repeatable record of why a resource received a particular evaluation.

But there are questions it cannot resolve through scoring alone.

A maintainer may know that a particular resource fills an important gap in the collection.

A new project may be unusually innovative despite having few adoption signals.

A resource may have historical significance even though active maintenance has stopped.

A commercial resource may provide exceptional value that outweighs concerns about openness.

A category may deliberately include competing approaches.

These are editorial decisions.

The purpose of ResourceRank is not to eliminate those decisions. It is to give maintainers a more consistent starting point for making them.

I think of the relationship this way:

ResourceRank structures judgment. It does not replace judgment.

One of the more unusual signals in ResourceRank is neutrality.

I included it because curated collections frequently operate in environments where inclusion has commercial or reputational value.

A link in a respected Awesome List can produce traffic.

A recommendation in a newsletter can produce customers.

Inclusion in a dataset catalog can improve discovery.

Placement in a learning library can create credibility.

As soon as inclusion has value, people have incentives to influence the curation process.

That does not mean self-submissions or commercial resources should automatically be rejected.

Creators are often the people most likely to know that their project belongs in a collection.

But maintainers should be able to distinguish between a resource being submitted for consideration and a collection being used primarily as a marketing channel.

ResourceRank can therefore consider factors such as self-submission and promotional language as part of the review.

The goal is not to punish commercial projects.

The goal is to protect the editorial independence of the collection.

That independence is part of what makes curation valuable in the first place.

Openness creates a similar nuance.

I work extensively with open source projects, so licensing and source availability matter to many of the collections I maintain.

But an open source resource is not automatically better than a proprietary one in every context.

A curated collection of productivity tools might reasonably contain both.

A dataset catalog might care deeply about licensing and reuse rights.

An open source software collection might make an approved open source licence a fundamental requirement.

A newsletter source list may barely care about software licensing at all.

This reinforces why ResourceRank uses configurable profiles rather than imposing a single universal definition of quality.

Quality depends partly on the purpose of the collection.

The important thing is that the criteria are intentional.

The workflow I envision is deliberately straightforward.

A maintainer gathers information about the submitted resource and places it into a structured JSON input.

They choose the review profile appropriate to the collection.

ResourceRank evaluates the input.

The maintainer then reviews the output, paying particular attention to the individual signals, strengths, concerns, confidence level, and category fit.

From there, the maintainer makes the final decision.

In practical terms, the process looks like this:

That final possibility interests me as much as the scoring itself.

If review outputs are preserved, maintainers can begin developing a history of curation decisions rather than relying entirely on memory.

This introduces another problem that ResourceRank may eventually help address.

Maintainers accumulate knowledge.

We learn that certain types of submissions tend to require additional scrutiny. We remember why a particular project was rejected two years ago. We know why one category has stricter standards than another. We recognize that a previously rejected project substantially improved and now deserves reconsideration.

But much of this knowledge disappears into pull request discussions, issue threads, commit histories, or the maintainer's memory.

Structured review records could provide another layer of institutional memory.

Over time, a collection could potentially answer questions such as:

This is where a simple scoring tool begins to intersect with governance.

The value is not merely producing a number today.

It is making the reasoning behind curation more legible tomorrow.

Potentially.

There are obvious opportunities for deeper automation.

A future version could retrieve GitHub metadata automatically, examine repository activity, analyze documentation, inspect licensing information, assess README quality or generate preliminary qualitative observations.

An AI-assisted layer could potentially help evaluate characteristics that are difficult to express through simple rules.

There is also the possibility of learning from historical maintainer decisions.

If enough structured review data existed, future systems could examine patterns across accepted, reviewed and rejected resources.

But I think there is an important sequencing issue here.

Automation should follow a clear evaluation model, not substitute for one.

Before asking an AI system to evaluate whether a resource belongs in a curated collection, we should understand what we mean by “belongs.”

ResourceRank v1 is intentionally focused on those foundations.

The sophisticated part can come later.

Another possibility emerges if structured reviews accumulate over time.

They become data.

Each review can contain the characteristics of the resource, individual signal scores, the automated recommendation and eventually the maintainer's actual decision.

Over enough reviews, that could become a dataset describing real-world curation decisions.

That dataset could help examine questions such as which signals correlate with acceptance, how different review profiles behave, where automated recommendations disagree with maintainers and whether particular criteria introduce unintended biases.

It could also provide training or evaluation data for future classification models.

But I think the most important field in such a dataset would be the human decision.

The goal should not be to collect enough examples that the maintainer can eventually be removed.

The goal should be to learn how structured systems can better support maintainers.

That difference matters.

ResourceRank also reinforces something I have increasingly come to believe about curated collections.

Curation is not one decision. It is a system of decisions.

A maintainer defines scope. They establish categories, decide what evidence matters, review submissions, determine what belongs, reconsider older resources, remove things, respond to changing standards, explain decisions to contributors and maintain the integrity of the collection over time.

If those decisions are entirely informal, the quality of the collection depends heavily on individual memory and consistency.

Some informality is healthy. We should not turn a small open source list into a bureaucracy.

But lightweight structure can make curation considerably more durable.

That is the space where I see ResourceRank fitting.

There is an obvious danger in building something called ResourceRank.

Once a number exists, people naturally want to optimize for it.

That is why I want to be cautious about treating the score as a definitive measure of resource quality.

A score is an abstraction.

It represents certain selected characteristics according to certain configured weights.

Change the characteristics, and the score changes.

Change the weights, and the score changes.

Change the context, and the meaning of the score changes.

There will always be qualities that are difficult to quantify: originality, usefulness to a particular community, historical significance, conceptual importance, unusual innovation or simply the maintainer's understanding of what their readers need.

ResourceRank should not flatten those qualities.

Its purpose is to make the measurable parts of a review more consistent, leaving the genuinely qualitative parts visible to human judgment.

That boundary is important.

So, can we measure the quality of a curated resource?

I think the answer is: Partially.

We can measure signals associated with quality, define criteria, make weighting explicit, identify strengths and concerns, distinguish strong evidence from incomplete evidence, evaluate fit within a particular context, make similar submissions easier to compare and record why a recommendation was made.

What we cannot do is compress every meaningful editorial consideration into a universal score.

And I do not think we should try.

The goal is not to transform curation into an algorithm.

The goal is to make curation more consistent, transparent, and explainable while preserving the judgment that makes it valuable.

That is the experiment behind ResourceRank.

My work on Awesome List governance started with a fairly simple belief: a curated collection should be judged by the quality of its decisions rather than the number of links it contains.

ResourceRank takes the next step.

If those decisions matter, perhaps we should become better at understanding how we make them.

Not so that software can become the curator.

So that curators can make better decisions.