Big Data in Agriculture - Mark Luckey - Storyboard Multimedia

As introduced. I'm Mark. I'm from Storyboard, multimedia software development company. If it makes you feel any better, I was a biochemist once upon a time, 2001, I started consulting to the Australian Crop Accreditation System. Since then, I've done an awful lot of offshore software development, launched a wholesale travel management platform and overseen the design, implementation and management of NVT online and through the process, get a lot more gray hair. Last week we launched the NBT app on the Android and Apple stores. Go to the store on your iPad, search for NBT and today I'm here to talk about not just big data, but big data for growers. So, Big Data, bit of a definition up there for you. Last month, US grower and agricultural lobbyist Mary Kate Thatcher said that data ownership and access is of great importance to growers. Trust is important. And for growers, what Big Data really condenses down to is what should I grow and how should I grow it? And there's a lot of correct answers to that question. But more and more the answers will lie in Big data. And it's something that as researchers and as tecos, we need to deliver. Broadly speaking, the value proposition of Big Data comes, I guess, in three layers. Simplifying so much in this topic, it's not funny, but there's three layers of value. The first one is efficiency improvements through better practice. The second one is through growing the value of data, which, or if you like, increasing its potential or its applications or its value to a number of people. And third is this weird concept of answering questions that we don't even know exist yet. One of the big concepts is convergence. As data converges, its value increases. As it fragments, its value decreases. And if you think of the greatest research done in the world published in a book that gets put up on a shelf and forgotten about, it becomes fragmented, it becomes isolated, it becomes forgotten. And it paves the way for that research to be repeated again and maybe printed in a book and put up on a shelf 10 years later. This has happened numerous times in our industry. It's a phenomenally inefficient way to manage research dollars and to manage data. Using my experience in mbt, I'm going to use that as a bit of a narrative to walk you through some of the important lessons and fundamentals and essential considerations of Big data. Now, the genesis, if you like, the start of what the National Variety Trials Program came from was A bloke called Lazenbeard 94 and 97 who identified considerable differences between states and organisations Regarding the release protocols for new varieties. Back at that time, organisations were breeding their own varieties, running their own trials and publishing their own results in their own glossy marketing brochures. The outputs were books. They were books that contained data that may be up to 18 months old. @ the time on the ACAS board, a grower called Hugh Roberts said to me that it might take anything up to 25 hours on the kitchen table with all the books making sense of the data available to them. Now, if you look at an output that might have come from around that period of time, something we obviously recreated, but this happened on the left. That's the full list of results from variety trials, all the varieties and how they performed. On the right is how the results from the variety trials appeared in the glossy marketing brochure of Company A. This happened all the time. Fundamentally not illegal, not actually incorrect, fairly misleading when you consider it. So at that stage in the industry, there are real challenges of accreditation process and accountability. So that's why ACAS was established, and ACAS was established in 97 to develop accreditation protocols for crop evaluation by public pruning programs for crop variety trials and to provide information on their performance to growers. It was about credibility. And at that stage, a programmer in South Australia was compiling all the information into an access database and put plainly, he failed. The great challenges at that time were politics and computer power. So things progressed. 2001, I got involved with ACAS. In 2002, we got a new high powered programmer, a fiery Eastern European with an IQ and a temper that was entirely off the charts. Give you an insight into how we were programming in 2002. I spent a lot of time with this guy. One Wednesday, about five o', clock, I went over to his place and we started work. We were really struggling to get all this state data onto a CD ROM so it could be looked at. Started about 5, not so long after his wife Svetlana brought us dinner. And we were fairly confident that if we got our heads into this we could finish it off that night. And, you know, we sort of worked through these iterative development and testing cycles, which was coding on one computer, coding, compiling an executable over to a zip drive, put onto another laptop for testing. And we went sort of on and on. We were consuming a lot of Svetlana's pastries, a lot of Alex's illegal Chop Chop cigarettes and just so much of that gritty Turkish coffee it wasn't funny. But we powered on, we were making progress and then Svetlana was serving us Lunch, dinner on Thursday night came around and we were pretty annoyed because we were sort of like really into it. We were that close. And we were fairly caffeinated by that stage as well. And the iterative testing cycle now involved 23 minute naps. Every time Alex compiled the software for transfer, we were finding that the caffeine wasn't working so well. About 7am Friday morning, a processor fan failed, something melted, the hard drive shattered itself and we realized that the word backup hadn't made its way into the 38 hours since we started. I drove home, Alex had a breakdown. We spoke about three weeks later, but we did eventually produce the Variety Selector CD rom. And it was exciting times. We had national data all in one place, data across borders. It was exciting and you had reputable systems and processes in the different states and you had these systems that couldn't talk to each other. A seven on one side of the border was not a seven on the other side of the border. The data was fragmented, modifying the language that Hugh Roberts, the ACAS board member, used. Poo in poo out. You can't polish poo fairly bit more colourful than that. We couldn't combine the data across the state borders and something clearly had to give. And what gave, sort of 10 years after Lazenby had identified these challenges in the industry was the national variety trials. No scope for bias, transparency, growers, reliable information, assess the claims. Today We've got over 5,000 GPS encoded trials. There's about 4 million data points, 20 million tiny tag temperature points. And it's astounding, however, how over a decade the questions and the comments and the expectations had changed. I was at grains work and I was pretty excited. I'm demonstrating this stuff to a grower. Great photo opportunity, 70 year old bloke, me, young and cocky and arrogant, showing off the software and told him about the 25 hours with the crop sowing guides and he said come search here, here and here for weeks. And we started downloading a customised data set for him so he could view and take home. And he said I made up at a 33k modem and I haven't got 5 minutes to wait for that to download. And he walked off 25 hours, down to a couple of hours and he didn't wait five minutes for the download and I just about had a breakdown at that point. So what we released last week was, Alrighty, go on play Video Play, yes. Was the app onto the Apple and Android stores. You can, you can download it. When you fire it up, it checks Back to the ACAS server to see if any new data has recently been released. It contains data at a national level point and click interface. It's all map driven. Your results are green for good, yellow for a bad average, red at the bottom for I won't say bad, but performing below expectations or varieties. Tabular presentation of results. Of course it's broken up into state level and agroecological zones on the fly. It's calculating this data as you click it gives you the results. You can go state level, national level, or you can even go down to a site level to process data. And it's doing it all on the fly on the maths developed by Sagi, which is one of the worst names for an agricultural research project ever, or the statisticians projects. But because the underlying and the underpinning standards and protocols and accreditation was now national, you could be a grower who lived across, near a state border, near regional borders and draw your own dynamic region and produce your own custom data set and thus enabling a grower to say what's going on near me? I don't really care state level what's happening in my area. Because that's the way growers think. They want to know what's going on in their local area. They can search, they can whip up some graphs, they can look at what's interesting, they can discuss it with their agronomists and if they want to, they can then email that off to their friends. So seven across the border equals seven. What took 25 hours can now take a minute. The data is up to date. A batch job runs every night to release new data. But it's astounding again how the questions and the comments and the dialogue around this all changed. We released this two days before Windows 10 came out and I was asked, is it compatible? So I don't know yet. We've already been asked if it will be compatible with the new iPad Pro which gets released in November. And it's astounding how the changes and the expectations really do evolve as things go along. Lets bring it back to the real world or the rest of the world or big data in your world. I'd like you to think Facebook, LinkedIn, Google, TripAdvisor, any one of a million apps you've got installed on your iPhone or your Android device that's tracking everything you do. Your data is a commodity. You're currently giving it away. A clothing retailer with the initials CR buys data from credit card companies to see what its customers do after they leave that store. They track where they're spending the money, what they're spending their money on to improve their product offering big data to try and get more money out of our pockets. And I'm a geek, I love technology, I love computers, I always have. But before we go enabling big data, before we start opening up these opportunities, we need to really understand who is going to exploit them and who is going to make the most of them, who is going to benefit. If I look back through 94 Lazenby ACAS in 97, GRDC implementing NVT back in 2003, I'm not sure if GRDC actually implements the value of the journey they began and where we are led to today. Because the implementation of NVT and where it's at today and the data it's at today has actually become a primer for big data in the grains industry. The challenge for someone like GRDC and other research organisations that we're all working for or with or getting our funding from is do they appreciate the data and can they allow it to further appreciate itself, you know, the 5,000 trials? That data is now firmly in the hands of Australian growers to assess, to review and to make informed decisions upon in an unbiased manner. And as a model, as a story, as an entire program, NVT is something that should and should be leveraged across a broader sector of industry. And so it does again come down to this concept of data, now again grossly simplified. I could put stacks more on there, but this is what we've been talking about as a data custodian model, where growers are placing their trust in the data custodian to make easy access to available, interesting and informative data. And if you like, in the middle is the guts of it. It could be. It's more than just variety trials, by the way. It's a whole stack of research projects. It's your standards, it's your naming conventions, it's the real power of what goes on and on the left and on the right, if you consider things like the data security, synchronization and user security, it's almost like a plug and play ideology and mentality about, well, there's standard ways to get data in and standard ways to get data out. What the question on the right asks is really, what else can you do with the data? Because that's where the value comes from. It's not just about crop variety data, it's like, what other insights can you draw from that? Who else will get value for that? And in turn value the research and the data, you're producing more. And so that's why I commonly refer to it as a big data primer and the app we've just produced, whilst it's the pointy end, if you like, of 5,000 data trials over the past decade and more and more as they come along, it's also just the beginning of something else. But all this joy of collaboration and sharing and so on, we've got to now come back to the real world, where the very nature of R and D is about competition, confidentiality, competitive advantage, and that all gets in the way of sharing. Because if individual projects are going to contribute to big data, the ideology is sharing. Now, sharing infrastructure, it's relatively easy to get your head around. We can all drive to work on the same road. Sharing data, though, starts to really scare people. Consider a company like GlaxoSmithKline, who spends about $4 billion to get a product to commercial reality. They, along with another half a dozen other drug makers, have released clinical studydatarequest.com they didn't spend a lot on the marketing for the name, but ClinicalStudyDataRequest.com now has 900 clinical trials and research projects that are underway that have been published to see who else can get value out of the data. One of the challenges is, well, pharmaceuticals can do it, why can agriculture and agronomy not so? One of the joys of running a company in the offshore development model is it's really hard sometimes getting to India. It's about 26 hours into end, so I spent Wednesday night nor day Thursday at Crown Plaza Singapore Airport, Changi Airport in Singapore with one of my development partners talking about strategy, wearable devices, emerging technology, staffing, politics and how we're bringing it together in our relationship later on. Google, Amazon, Echo, that's going to be one of the next big things, by the way. And he shared with me a story of a very excited client who rang him up and said, I want you to come and install big data. We need it. And he said, great, what are you planning to do? And he said, well, we're not really sure yet. He said, that's cool, because often the questions come once you've got the data. So what data have you got? He said, well, none. He said, well, where are you going to buy it from? He said, well, what do you mean? And he said, well, what are you intending to do with big data? He said, I don't know, but I want you to come and install it. Obviously a gross misunderstanding of what's needed to do before you progress. But the thing is it's here but it does need to be understood and it doesn't need to be just understood by researchers. If you look at the political level, there needs to be greater support of the science, but the politicians need a reason to support your organisations like the GRDC really need to be the custodians, the ones who provide the framework, the the ones who give the vision on what they're going to deliver. Coders like myself or the non science geeks, the programming geeks. We need to become more generalists and help work across those vertical silos of research programs to bring it together. And researchers really need to start implementing a big data mindset into everything they do. With all the presentations I should be saying where to from here? Have some fun with it to start. Don't start off non academically. Go and buy Freakonomics. It's a book released by some guys from the States which is just phenomenal what they've done with the data I think. Embrace it because it's here to stay. Make it part of everything you do and everything you review. Consider the big data context of it because the big data right is here and at this point R and D has a great chance to be a driver of it. Thank.