As a photography enthusiast, this makes me cringe. I'll acknowledge that the samples they show came out fine, but I suspect that in the real world this won't work well as a generalized solution.
There are a lot of things that if you get wrong, you'll ruin the image. This is true from an artistic perspective, but even with a lower bar, it's very easy to crop an image to make the content distracting or disturbing.
There are rules of thumb, for example, about how to crop images with people. It's not just that you want to preserve the faces. You need to be careful about where on their bodies you crop: if you do it right at a joint, it will tend to look like the person is an amputee. So you've got to crop a leg at mid-thigh rather than at the knee, for example.
There's also an idea of "look space". If you crop so that the direction of the subject's gaze is toward the outside of the image (rather than toward the middle), the subject will tend to look crowded and the overall composition unbalanced.
And there are overall compositional rules, like rule-of-thirds, leading lines, etc. Cropping an image is likely to damage these, thus harming the aesthetics of the picture.
I won't make the claim that this stuff is impossible to automate (the very fact that I can list these rules of thumb suggests that much of it can handled without creativity as such), but doing so in a way that preserves the desirable qualities of a good image will require a LOT more sophistication than what's being brought to bear here.
If you're interested in preserving composition using automatic methods, you may want to check out the field of image retargeting. Instead of cropping, it attempts to warp 'non-important' parts of the image to reach the desired aspect ratio. An implementation of it in photoshop is called 'content aware resize', but I haven't seen an evaluation of it against the state of the art research.
Some time ago I tried to figure out the dumbest way to find important features in most photos given a certain resolution.
There's face/features detection (and other CV) in here, too, but you can get by with some really cheap tricks in a lot of cases. In fact, the meat is it finds things that are chromatically different or more, uh, feature-full.
> Square pegs don’t fit in round holes, but what if you have power tools?
Plastic fumes :(
> But it doesn’t have to be this way! In this post, we present a technique that we use for intelligent cropping: a fully automatic method that preserves the image’s content
This seems like overengineering. Why not just build a UI to quickly crop a set of images exactly as desired, with a scalable box given the required aspect ratio?
Since it's likely you'll want to verify these kinds of images before posting them anyways, and might go back and fix some later, it seems like you're not really cutting down on anything other than maybe a few seconds per image to move that initial bounding box.
For advertising, sure; each ad campaign has its own dedicated staff to do some busywork.
My own use-case for this, though, involves ~500k images (think company logos appearing in profiles like LinkedIn, fit to various layouts.) I certainly couldn't do that (or even verify that) by hand.
I have had the opportunity of testing shots with & without faces, the conversation rate is way higher when faces are included.
In the past, cropping to remove faces was done for costs reasons. The retailers paid the model for printed use only and had to pay extra to use them on ecommerce websites.
We ran A/B tests, paid the models rights for ecommerce usage and demonstrated a steep increase in conversion.
The rights issue is the reason some companies provide "paste a face on pictures" automated services. Basically you buy the rights on a few headshots and the service stitches the faces & clothing together (see OpenCV stich functions...).
The problem is that while decent, the services often end up in the uncanny valley, actually decreasing conversion.
It's not a universally great way to crop images while preserving the most context. In my early attempts relying on face detection heavily led to some very poor crops.
You should use them as centers of gravity, measure normalized size, compare against other features, etc.
There are a lot of things that if you get wrong, you'll ruin the image. This is true from an artistic perspective, but even with a lower bar, it's very easy to crop an image to make the content distracting or disturbing.
There are rules of thumb, for example, about how to crop images with people. It's not just that you want to preserve the faces. You need to be careful about where on their bodies you crop: if you do it right at a joint, it will tend to look like the person is an amputee. So you've got to crop a leg at mid-thigh rather than at the knee, for example.
There's also an idea of "look space". If you crop so that the direction of the subject's gaze is toward the outside of the image (rather than toward the middle), the subject will tend to look crowded and the overall composition unbalanced.
And there are overall compositional rules, like rule-of-thirds, leading lines, etc. Cropping an image is likely to damage these, thus harming the aesthetics of the picture.
I won't make the claim that this stuff is impossible to automate (the very fact that I can list these rules of thumb suggests that much of it can handled without creativity as such), but doing so in a way that preserves the desirable qualities of a good image will require a LOT more sophistication than what's being brought to bear here.