Inspired by Van Gogh, the company’s engineers are now using algorithms to mimic creative practices and sell modern art