Why Vision Inspection Works on the Pilot Line and Fails on the Second Shift

An automated inspection system is unusual among industrial projects in that it usually works immediately. You collect a few hundred images, train or configure a detector, run it against the line, and it finds the defects. Then it goes into production and the false-reject rate climbs, an operator switches it to advisory mode, and within two months it is a camera nobody looks at. The reasons are consistent, and almost none of them are about the model.

The first and largest is lighting. A vision system does not see a part; it sees the light reflected off a part, and every property you care about — contrast between a scratch and its background, whether a surface reads as matte or specular, whether a dark defect on a dark material is visible at all — is a property of the illumination rather than the camera. A pilot built under whatever light happened to be in the cell will behave differently when the roller door opens on a sunny afternoon, when the overhead fluorescents are replaced with LEDs at a different colour temperature, or when a lamp ages and drops output by fifteen percent without anybody noticing. Enclosing the inspection station and controlling its light is unglamorous and is usually the difference between a system that survives and one that does not.

The second is that the pilot dataset is not a sample of production. Images collected over two weeks in spring contain one set of materials from one supplier, one range of ambient temperature, one operator's setup habits and whichever defects happened to occur. Production contains the second supplier whose surface finish is slightly different, the winter batch, the night shift, and the defect class that appears four times a year. A detector trained on the first will treat much of the second as anomalous, and the failure looks like the model degrading when in fact the world simply got bigger.

The third is class imbalance, which is inconvenient rather than subtle. A line running at any decent yield produces very few defects, so a dataset gathered by running the line is overwhelmingly good parts. That makes accuracy a useless metric — a detector that passes everything scores 99.5% — and makes the rare classes the ones you have least evidence about, which are also the ones that matter. The practical answers are to deliberately collect and preserve defects rather than scrapping them, to keep a physical library of known-bad parts for revalidation, and to measure with precision and recall on the defect class rather than with overall accuracy.

The fourth is that nobody agreed what a defect is. Ask three inspectors to grade the same fifty parts and they will disagree on several, usually the marginal ones, which are exactly the parts the system will be asked about. If the labels are inconsistent, the ceiling on model performance is set by that inconsistency and no amount of training gets past it. Running a short agreement exercise before any modelling — same parts, several inspectors, measure how often they match — tells you what the achievable number actually is, and often produces a more useful specification than the original one did.

The fifth is drift, and it is the one that turns a working system into an abandoned one. Cameras move slightly when a fixture is knocked. Lenses collect dust. Suppliers change. Products get revised. Without a scheduled revalidation against known parts, the first sign of trouble is an operator's judgement that the system is wrong more often than it used to be, which arrives long after the drift began and destroys trust in one step. A weekly pass of a golden sample set through the station, with the results logged, costs a few minutes and gives you a curve rather than an opinion.

What follows for design. Control the light and enclose the station. Fix the optics and the working distance mechanically so they cannot be nudged. Collect production images from the beginning, including the boring ones, and keep the defects. Get labelling agreement measured before you promise a number. Decide the operating point deliberately — inspection is a trade between false rejects and escapes, and which way to lean is a business decision, not a technical one, so make somebody own it. Log every decision with its image, because a system whose judgements cannot be reviewed afterwards cannot be improved and will not be trusted.

And keep the human in the loop on purpose rather than by accident. The strongest deployments do not replace the inspector; they hand the inspector the parts the system is unsure about, which is a smaller and more interesting job than looking at everything, and which produces exactly the labelled marginal cases the next model version needs.