Robyn Anderson tested ChatGPT's new Operator tool and found it can't log into password-protected sites or solve CAPTCHAs. It handles simple tasks like booking vacations and ordering food, but fails at complex business workflows. She recommends waiting before buying.
Chapters
So you asked if the ChatGPT operator can scrape data from a website that you need to log into. And the answer is a big fat no. You can't log into anything that needs a password at this point because the operator can't pass any sort of CAPTCHA. It asks you to go into takeover mode, and that needs a lot of handholding, which kind of defeats the purpose. So I've been playing with it. I was up late last night trying all different scenarios, and the scenarios you see that sold me on it were all about like planning vacations and ordering food and things like that. And so far, when I ask it to actually do things that would help run my businesses, it's not completely, completely automated. It's doing cool things and logging in and working, but it's not completely doing a task 100%. So I would say hold off if you're thinking of buying it. And actually, I'm going to spend like the next week trying a bunch of different things, and I'll report back on whatever is successful. But so far, a lot of flops.