Beau Brummell's contribution to English life was subtraction. He took away the colour, the embroidery, the scent and nearly all of the jewellery, and left behind a cut, a cravat and an insufferable superiority of manner. It is among the better jokes of cultural history that the word he bequeathed has come to mean something close to the opposite — the gentleman in plum velvet with the buttonhole and the stick, whom Brummell would have crossed Bond Street to avoid. The dandy has been misconstrued almost from the beginning, and the misconstruction has outlasted the original.

Rose Callahan and Nathaniel Adams have spent the better part of twenty years inside that confusion, and their achievement is to have treated it as a subject rather than a joke. She began photographing well-dressed men in New York in 2008; he arrived in 2010 with a book proposal and a research trip to the Congo behind him, and the two discovered they had been documenting many of the same people from opposite directions, she with a camera and he with a notebook. I Am Dandy followed in 2013, fifty-seven subjects, a preface by Glenn O'Brien. We Are Dandy came three years later, wider by two continents and ten countries, with Dita Von Teese supplying the introduction. This third volume arrives now, and anyone with an ear for titles should stop at the pronoun.

I, then We, and now They. The first was a declaration made by each sitter in turn. The second was a claim of common cause. The third steps outside the group and looks at it, which is either the mark of a project that has grown too large for its authors to speak from within, or the moment a study becomes properly anthropological. The subtitle makes the same move less subtly: Gentleman has become Gentleperson, a coinage that will set certain teeth on edge and is nonetheless doing real work. The publishers say the book is concerned with expanding who a dandy can be. That is a promise with a hazard attached, and it is the interesting thing about the volume.

The cover states the case before a page is turned. A cream linen jacket, a floral tie held by a clip, a white shirt, a hand laid across the chest wearing two signet rings and a band of amber — and burgundy varnish on the nails. Every element save the last would have satisfied a Jermyn Street window in 1935. The last rearranges the whole. It is a good photograph and an efficient argument, and it tells you that the terms of the series have altered.

Whether that alteration enriches the category or dissolves it is the question the book has to answer. There is a respectable case that the dandy was always defined by exclusion — that Brummell's severity, the labour disguised as ease, the refusal of the crowd's judgement, cannot survive being extended to everyone who dresses with intent, and that a term meaning anybody eventually means nobody. There is an equally respectable case on the other side: that dandyism was never a costume but a posture of refusal, that the people most obliged to invent themselves in public have always been its truest practitioners, and that the earlier volumes were narrower than their subject merited. Callahan has said in recent years that her sitters are frequently gender non-conforming and queer, and photographs them accordingly. The third book looks like the point at which the series stops apologising for that and builds on it.

What has kept these books above the level of the style supplement is method. The sitters are photographed where they live — in their own rooms, among their own objects, in the settings they have arranged for themselves — and Adams's texts run long enough to admit contradiction, routine and expense. Street-style photography takes the surface and departs. This takes the surface as evidence and then goes indoors, which is the only way to find out whether a man's clothes are an argument or a hobby. On the showing of two previous volumes, the promise on the jacket is generally kept.

For members of this Club the interest requires no explaining. Dandyism and eccentricity are cousins rather than twins — the dandy edits, the eccentric accumulates — but they keep the same hours and drink in the same rooms, and a survey of people who have decided to look like nobody else is bound to catch a number of our own. Both authors have been guests of the Club and are known to a good many of us, which is stated plainly at the foot of this notice for the reasons such things are stated.

At £60 it is not an impulse. It is a large, well-printed object of the kind Gestalten has been producing since before the fashion for them, and it will be looked at rather than read through, which is what the form is for. Buy it for the pictures and stay for the texts, which are better than the pictures deserve, if that is not too backhanded a compliment to a photographer of Callahan's standing.

Brummell died in Caen, mad and in debt, having been mocked in the street for the state of his linen. He would have detested the nail varnish and understood exactly what it was doing there.

Declaration of interest: Rose Callahan and Nathaniel Adams have both been guests of the Eccentric Club, and members of the Club appear among the book's subjects.

They Are Dandy: Portraits of the Modern Gentleperson (Gestalten, 288pp, hardcover, £60) is published this week.