← The Thread: notes on shopping an outfit from a photo

How it works

Why does reverse image search fail on clothes?

A reverse image search looks for other copies of your picture. That is a different question from what the person in it is wearing, and it is why the results feel so useless.

Save to Pinterest

You screenshot an outfit, run a reverse image search on it, and get back nine blog posts that used the same photo. None of them sell the coat. This happens every time, and it is not a bug.

A reverse image search, meaning a search for other pages that used the same picture, is doing exactly what you asked. You just asked the wrong question.

What question does a reverse image search answer?

It answers: where else does this image appear?

That is a real and useful question. It finds the original source of a photo. It shows you where a picture has been reposted. It catches a stolen image.

What it cannot do is look at a person and tell you what they are wearing, because it indexed the picture, not the garment.

What question were you actually asking?

You asked: what is that coat, and where can I buy one?

Answering that needs a different first step. Something has to look at the photo and decide "that is a wool overcoat, camel coloured, double breasted, hitting below the knee". Only then can a shopping search happen, because shops are searched with words.

Snagfit does that first step. It reads the photo, names each garment, and writes a short description of it. The description becomes the search. Your picture is not part of the shopping search at all.

Is visual match search any better?

Some tools go further than plain reverse image search and match visually similar pictures rather than identical ones. Google describes Lens as comparing objects in your picture to other images and ranking them by similarity and relevance.

That is a real improvement, and it works well in one specific situation: a clean product photo of a current item that appears on many retailer pages in a similar form.

It gets much harder when the garment is on a person, at an angle, in a room, in motion, partly hidden. Which is to say, in every screenshot anyone actually takes.

Why does describing the garment work better?

Two reasons, and the second one is the bigger of the two.

A description is specific in the ways that matter. "Cream chunky retro runner sneaker with suede overlays" filters out thousands of sneakers that happen to be photographed against a similar background. Visual similarity does not know which parts of the picture you cared about.

A description survives the item being unavailable. This is the important one. If the exact coat sold out two seasons ago, there is no listing image for a matcher to find. But "camel double breasted wool overcoat" still matches coats that exist right now. You get the closest thing you can actually buy rather than an empty screen. There is a longer post on why close matches happen.

Why does the name reverse image search mislead people?

Part of why people expect more from it is the phrase itself. "Reverse image search" sounds like it should be the opposite of a normal search, so if a normal search takes words and returns images, the reverse ought to take an image and return words.

It does not. It takes an image and returns more images, plus the pages those images sit on. The direction that reverses is where the query comes from, not what you get back.

Once you read the name that way, the results stop being surprising.

When does image matching break on clothes?

It is worth being specific about this rather than saying "it does not work well". Image matching fails in four identifiable ways, and every screenshot you take hits at least two of them.

The garment is on a body. A product listing photographs a coat flat, or on a mannequin, front on, evenly lit, against a plain background. A screenshot shows the same coat on a moving person, half turned, creased at the elbow, with a room behind it. Those two pictures share a garment and almost nothing else visually.

The garment is partly hidden. An arm crosses the chest. A bag strap runs across the front. Another person stands in the way. A human still reads the coat correctly, because a human knows coats continue behind arms. A pixel comparison does not get that for free.

The lighting has changed the colour. Indoor light, a colour grade on a video, a filter, a screen at low brightness. A camel coat under warm light and the same camel coat in a listing photo can be measurably different colours.

The item is not in the index at all. No listing exists, so there is nothing to match. This is the one that no amount of technical improvement fixes.

Why does a description survive all four cases?

Look at the same four cases with a description-based search.

The garment being on a body is not a problem, because the model is reading what the garment is rather than comparing shapes. A camel double breasted wool overcoat is a camel double breasted wool overcoat whether it is flat, on a mannequin, or on someone walking.

Partial occlusion degrades the description rather than breaking it. If the model can see the collar, the closure and the colour but not the hem, it writes what it can see, and a less specific description still runs a working search.

A colour shift is a real problem for both approaches, and it is the one place where you should take the fix seriously: pick a brighter frame.

And the fourth case is where the two approaches genuinely diverge. When no listing exists, image matching has nothing at all. A description still matches similar garments that do exist, which is the difference between an empty screen and a shortlist.

What does similar mean to each kind of search?

This is the part that explains why the results feel so different.

For an image matcher, similar means the pixels resemble each other. That sometimes produces genuinely strange results, where a returned product shares a background, a pose or a colour block with your photo rather than sharing the garment.

For a description search, similar means the words overlap. Search for "chunky cream retro runner sneakers" and you get shoes described in those terms by the people selling them. The failure mode is different too: instead of an unrelated product that looks like your picture, you get a related product that is not quite the one you wanted.

Most people find the second failure more useful, because a near-miss sneaker is still a sneaker you might buy.

When is a reverse image search the right tool?

None of this means the technique is bad. It means it answers a different question, and there are times when that question is the one you have.

Use it when you want to find the original source of a photo, when you want to know whether an image has been reused elsewhere, when you suspect a listing has stolen a product photo from the real brand, or when you are trying to date a picture by finding its earliest appearance.

That last one has a practical shopping use. If a listing's photos turn up on another brand's site from two years earlier, you have learned something important about that listing before you enter a card number.

How can you test the difference yourself?

Take one screenshot of a person in an outfit and run it two ways.

Put it through a reverse image search first and read what comes back. Count how many of the results are pages that reused the photo, how many are visually similar images of something else entirely, and how many are actual product listings for the garment.

Then run the same screenshot through a description-based search and read the garment names it returns. Do not tap anything yet. Just read them.

The second list tells you what the clothes are called. The first list tells you where the photo has been. Both are answers. Only one of them is the answer you wanted when you took the screenshot.

Doing this once with a photo you actually care about is more convincing than any explanation, and it takes about a minute.

What should you do with a photo you care about?

Run it through something that names the garment first. Take the names it gives you and keep them, because those words are the reusable part.

Once you know the item is called a camp collar shirt and not a "shirt with a weird collar", every other search you run gets better. That includes plain Google, the retailer's own site search, and a resale app.

The photo was never the searchable thing. The words are.

Common questions

Why does reverse image search not work for clothes?

Because it answers a different question. A reverse image search looks for other pages that used your picture, or for pictures that look like it. Neither of those tells you what garment the person is wearing, so the results are pages that reused the photo rather than places selling the coat.

What is the difference between reverse image search and outfit search?

Reverse image search starts and ends with the image. Outfit search identifies the garment first, writes a description of it, then searches shopping listings using that description. Snagfit works the second way, which is why it returns garments with names rather than pages that used your photo.

Does Google Lens do a reverse image search?

Google describes Lens as comparing objects in your picture to other images and ranking them by similarity and relevance. That is image-to-image matching, which is closer to reverse image search than to a description search, though Lens can attach shopping information when a matched image is a product listing.

Why do reverse image results show the same photo on other sites?

Because that is exactly what it was asked for. If a creator's photo was reposted to twenty blogs and three aggregator sites, those are the closest matches in the index. The search is working correctly and answering a question you did not mean to ask.

Does reverse image search work better on a product photo?

Yes, much better. A clean product photo on a white background often appears on many retailer pages in near-identical form, so image matching lands on real listings. It struggles on a person wearing the garment in a room, at an angle, in motion, which is what almost every real screenshot looks like.

What should you use instead of reverse image search for an outfit?

A tool that names the garment before searching. Snagfit reads the photo, describes each item in words, and runs that description against current shopping listings. Even if you never tap a buy link, having the garment named lets you search for it anywhere else, since the words are the part that transfers and the photo never was.

Can reverse image search find a vintage or sold-out item?

Rarely. If the piece is not listed for sale anywhere, there is no product image for it to match against. A description search still returns similar garments, which is a more useful answer than nothing, since it matches on what the garment is rather than on whether a matching picture exists.

Is reverse image search ever the right tool?

Yes, when your question is about the image. Finding the original source of a photo, checking whether a picture has been reused, or tracing where an image came from are all reverse image search jobs. Identifying a jacket is not, since the garment was never what it indexed in the first place.

Sources

Keep reading

Comparisons

Snagfit vs Google Lens: which should you use?

They work in opposite directions. Google Lens compares your picture to other pictures. Snagfit describes the garment and searches for the description. That difference decides which one helps you.

7 min read
How it works

Why do two people in a photo confuse a search?

Group photos are the most common upload that comes back wrong. The search reads every garment it can see and has no way of knowing which person you meant.

8 min read

← Back to Snagfit