Bin picking and machine vision

Citation

Shafi, A. (2011), "Bin picking and machine vision", Industrial Robot, Vol. 38 No. 2. https://doi.org/10.1108/ir.2011.04938baa.002

Bin picking and machine vision

Article Type: Viewpoint From: Industrial Robot: An International Journal, Volume 38, Issue 2

The relevance

It is appropriate to begin with two facets of the natural world and our human life.

First, the world is filled with entropy, or disordered order, or alternately, ordered disorder. As an example, no two trees are identical yet trees of the same kind look similar. In the autumn, leaves do not fall in a rectangular grid pattern; they look similar but they are uniquely different and when they fall, they fall in a random order driven by the forces of wind and gravity. In a similar manner, it is natural to have parts made in a stamping machine or a die cast to fall in a heap in a bin in random order. By virtue of gravity, they do not fall in an ordered geometric pattern. Thus, entropy and randomness are integral in our natural world.

Second, we humans deal with this disorder of picking leaves off the ground, or separating clothes in a heap by two of the most used of our five senses: sight and touch. We are able to:

• •

  see things;

• •

  figure out where they are; and then

• •

  tell our arms, hands and fingers how to acquire them and perform an action.

This, too, has a mechanized analogy. It costs a significant amount of energy and money to bring order to unordered collections of objects. The science and art of seeing objects in a disarray with machine vision cameras and to figure out where they are in 3D space, and then to command robots to acquire them and perform a subsequent action is called “Bin Picking”.

The difficulty

Ever since robots became practical to use in factories, some three decades ago, people have seen the numerous bins in factories and hoped and wished and worked to make robots see these random objects and pick them in an automatic manner. Theoretical books have been written, some of our best universities have made serious efforts, companies have attempted, and called this goal the most difficult yet most rewarding of all robot challenges in factories and also termed it “The Holy Grail”. In 25 of those past 30 years, technology and the complexity of the task in geometry, lighting, shadows, and occlusion have evaded success and frustrated a lot of attempts. But just as most universities have turned their attention from bin picking in the 1970s and 1980s to networking, the internet, and security applications in the 1990s and 2000s, a handful of companies in industry have struck the beginnings of a gold mine in bin picking success, swept forward with a confluence of enablers to make bin picking not only possible, but practical, and less fearful of unreliability in factories. And, today, in 2010, dozens of successful bin picking applications point the way to solving the problem completely by 2020, a number synonymous with perfect vision, if not before. This is my conviction and prediction.

The breakthroughs

Difficult as it may be, bin picking does not suffer from any known barriers to computing or provable barriers to solutions, e.g. the infamous “halting problem” in which proofs exist that show that it is impossible to know if a general purpose computer program will stop or not in the next iteration of its execution. Mercifully, and in addition, the following enablers can be enumerated for bin picking: improvements in programmable lighting, higher resolution cameras, faster acquisition boards, faster computation, better algorithms, elasticity in recognition algorithms, a collection of 3D mathematical and matching, filtering and inference tools, and of course, a mature community of project managers who know how to measure variation, manage lighting and carefully select the solvable versus the still impossible and implement with confidence with measured returns in investment, and financial performance.

The confidence

There is no magic bullet to solving all of bin picking with one super algorithm. Rather a collection of algorithms have emerged that have progressively solved an increasing number of classes of geometries, arranged in a progressively increasing level of randomness. The appeal and growing confidence, in the last five years, has been in seeing a growing set of successful installations and subsequent citation in industry press and demonstrations in trade shows. More innovation is on the way, spurred by internet connectivity, more powerful computation libraries in open source or from proprietary suppliers, and of course, the motivation of financial benefit derived through overcoming entropy in our natural world.

Adil Shafi,Advenovation Inc., Houghton, Michigan, USA