Improvements in the Quantity Framework

My name is Nick Laraveer, I am a kernel developer here at Wolfram Research, and I'm also the lead developer of the Quantity and Units framework for the Wolfram language. And today we're going to be talking about some improvements that we've been making in recent versions to the framework as a whole, and then also have fairly targeted discussion talking about some of the recent changes we've made in the way that temperature units in particular have been updated and improved. As I said, we're basically going to be talking about improvements and optimizations made to quantity and the related functions of Wolfram language. Specifically, we're going to talk about the improvements in the general unit checking, compatibility and dimensionality framework, as well as conversion and and arithmetic, and then move on to discussing the specifics of handling temperature and temperature difference units in Wolfram language. So I will assume most people here, at least vaguely familiar with the quantity and units framework. But as a brief primer, a physical quantity as a concept is basically a material or systemic property that's quantifiable in some way, so we can measure it, such as the mass of an object, the length of an object, the time taken over a duration or ambient temperature change in temperature. In the Wolfram language, our canonical way to represent that is with a quantity expression which has a value or magnitude, and a unit. The unit expressions can be anything from a simple string such as meters, to an arithmetic combination of strings like meters squared per second or feet squared per second. Also mixed units like hours, minutes and seconds together as a single quantity, or various qualified versions of that, such as dated units for historical currencies and independent units for units that don't fit within the normal unit framework. But maybe something like accounting units. If I want to know how many cows there are per acre. For the most part a cow is not a convertible unit or a convertible quantity, but it's nice to be able to convert from acres to square meters, etc. So the quantity framework is supported throughout a lot of the Wolfram language. We're always kind of expanding areas that it interfaces with, but the three fundamental operations that we need to do any sort of work with units are validation, which is checking the unit, and the quantity part itself, dimensionality checks, which include things like looking at the physical dimension of a unit and also checking compatibility between different quantities and different units, and finally conversion. So it'd be functions like unit convert, currency, convert common units, where we're changing from one specific unit into another one, and all three areas are things that we've been optimizing in the last few versions. And for version 14, we'll have basically a streamlined version of all those fundamental unit operations in Wolfram language. So we'll kind of run through each of them a little bit on how much things have improved and why those improvements are non trivial to introduce. So our very most fundamental operation is validating units, saying, is this expression a valid unit? That is something that we've made considerable improvements in that. This chart is showing the evaluation time in microseconds for validating different kinds of units. So for a basic unit, we're at about a tenth of a microsecond, which is just about as fast as we can get for evaluation time. The actual validation is even faster than that. But anytime we call the evaluator, there's some overhead. So the benchmark difference here is showing there's been a 480 times improvement in speed for basic unit checking. Physical constants were a little more complicated. Compound units are combinations of units, but we're generally in the order of two orders of magnitude improvement across the board for saying whether something is a unit. So I won't go into the implementation details on everything, but I want to kind of use this as a test case of why this sort of lookup isn't super trivial. So when we're checking for why whether something is a unit for something like meters, it's pretty straightforward. Is this string a known unit string for more complicated units? You know, this is not a common physical quantity that one runs across, but it is a valid combination of different units raised to different powers. Sometimes they may be fractional powers or real powers. There may also be numeric constants in there. And just as kind of a basic test case, this is a function that generates a random string of somewhere between 4 and 20 characters. And if we want to model a basic version of unichecking, we can say, okay, let's take A list of 1,000 characters, or 5,000 units is about what we have in the Wolfram language. If we were to generate a list of 5,000 of them and then generate a new random string and say, is this one of the members of that 5,000 element list? It takes us about 10 microseconds to do that evaluation. So that is three orders of magnitude slower than what we're actually looking at. That's basically the amount of time it takes to run the function printable ASCIIQ on that given string. So just checking whether those characters are actually printable ASCII is a pretty simple operation, but it still isn't fast enough for what we want to do within the unit framework to keep everything moving along really quickly. So as a. Let's jump back down to where we were. So just as a test case here for how we've done this in known unit queue, if we look at the evaluation time for doing the same thing with a random string, we are down under a tenth of a microsecond to do that evaluation. That is nearly the same evaluation time that is needed to check whether that expression is a string. So again, this is kind of our most fundamental step in unit framework. We need to know whether something is a valid unit to process it, and we've made it basically as fast as we can with working with expressions. The next level of improvement is working on our fundamental evaluations once we've done validity, and that's checking compatibility and unit dimensionality. Here again, we've gotten one to two in a couple cases, closer to three orders of magnitude improvement in both checking what the unit dimensions of an expression are and whether two units are compatible. The basic way this works is if I ask for the unit dimensions of a unit like feet, I get this list of different unit dimensions and the powers that are associated with it. So unit dimension is a length unit raised to a single power. It's a measure of length, not of area or volume. Same goes with units that have compound unit dimensions one, because there is effectively a set number of physical dimensions, we can also represent this instead of as a list of the dimension and the powers of, just as a vector of unit powers. So for feet, I basically just know that there's a single unit dimension that's raised to power one for PA. I have three different unit dimensions of length, mass and time in a specific combination. Ordering here is important. What I basically have is a set list of keys for what the positional arguments are here. And we can dig into and basically say, if I want to look up the unit dimensions, I can just get the unit vector, check the corresponding keys that exist here, and then raise them to the appropriate power. By making those improvements, we have basically been able to improve compatibility checks too, because effectively compatibility is checking for a dimensional compatibility. So I can take the same information and say, okay, I want to know what this unit dimension vector is for a given unit. And even for compound units, I basically can do a substitution of operations. Because we're working with powers here, we're actually replacing the times operation with plus and the power operation with times. We're reducing the power by one. But I can combine all the individual unit Vectors for these sub expressions into one and see that the unit vector is the same, and that's effectively what compatible unit Q can do. Now. So just working with in most cases integer values. But there are rational and real valued exponents in some cases too. But it makes it a lot easier when we know we're basically just checking powers. And that's where a lot of the improvement in those dimensionality checks go. Unit conversion is the third operation that's pretty fundamental for working with quantity and units. For version 14, we have overhauled the way that unit convert works and we are again in the one to two orders of magnitude improvement. This is our main non trivial operation on units. So we've gotten a factor of 10 improvement for the most basic cases. And as cases get more complicated, we're in the range of 2 order of magnitude. So basically how conversion works is we need to know three things. We need to know the fundamental value or the scale the unit uses, what its zero value is and what its base unit is. And so for conversion to base units, we basically need to look that up for whatever the input unit is. So if I pull that information out about the unit feet, I can see that it's, this is the scale that it operates on versus meters. It has a zero value of zero meters and its base value is meters. So if I want to convert feet into meters, I basically need to multiply whatever my magnitude is by the scaling factor and add the zero value, which in this case is zero, and I get a ratio of feet to meters for our scale. Multiply in against our magnitude and we get 381 over 125, which is exactly what you get if you call unit convert. And that's effectively what it's doing under the hood. The operation is the same for non zero valued units. So degrees Celsius has the same scale as Kelvin in terms of a scaling factor, but has a different zero value. So we take the magnitude, multiply it by one and then add our offset from Kelvins. And that's fundamentally the operation that unit convert is doing to convert to SI base for basically any unit as it needs to look up those three values. And then it needs to do some basic arithmetic for converting between one unit and another. We could convert to our base value, our base units, and then convert again to a target unit. But it's actually more efficient if we look up those conversion factors for two units. So say I want to convert from degrees Fahrenheit into degrees Celsius, I basically need to look up what the zero value for each of those is what the scale is. And then combine the operation into a single evaluation here, where we're basically multiplying, we're subtracting a constant out after doing similar bits of arithmetic, where we're adding an offset and a scale. And that lets us do the same sort of conversion operation that unit Convert is doing. And there's some improvements, like we have some caching involved and other things that help keep things moving along. But these are the sort of fundamental steps involved in a lot of the unit arithmetic that we've been able to migrate over into an optimized form so that everything that's built on top of the basic operations in units is a lot faster now, even without needing to do any individual retooling for functions. But there are some cases where we have done additional retooling to make those functions even faster. The other big area of improvement is how we handle the difference between temperatures and temperature difference. Temperature is a special case because the colloquial way we measure ambient temperature is to use a scale like Celsius or Fahrenheit, and those have an arbitrary choice of zero values. So for degrees Celsius, it is the freezing point of water. For degrees Fahrenheit, it is decided traditionally based on a specific set value. But there are implications to the way that arithmetic gets handled when you have to account for those arbitrary zero values. Previously, how we handled this was we just had two different unit dimensions. We said this is a physical quantity of ambient temperature, or of absolute temperature as we call it. And this collection of units is a measure of unit of temperature difference units. And that created some problem because the interaction between the two of those wasn't super clear. So what we've done late on in version 13 was to unify them to a single unit dimension, but add some specific parameters for how those units interact. So just as there's a couple of basic primers here about what the difference between a temperature and a temperature difference is, but sometimes it's a little more clear to look at examples. So conversion, as I said, is one of our basic non trivial operations. And they're specific rules about how conversion is handled between temperatures between temperature differences and the interaction between those two. So when you convert one temperature to another temperature, you can go from any scale to another. I can go from degrees Celsius to Fahrenheit or vice versa. I can convert to Kelvin. Kelvin is kind of our constant reference point because it can be used as an absolute temperature or a difference of temperatures. It has a fundamental zero value of absolute zero. And so that's our reference point. We can convert between temperature differences. Again, temperature differences also have a zero value of zero and they represent the difference between two absolute temperatures. So I can convert from degrees Celsius difference to degrees Fahrenheit difference. You'll notice the difference units have this small delta under the degree sign, indicating it's a measure of temperature difference. And again, Kelvins can be used on either scale or in either fashion of absolute temperature or temperature difference because of their it's fundamental zero value. It is possible to convert from a temperature to a temperature difference. When we do that, we are implicitly assuming that that temperature difference you're asking for is a difference against absolute zero. So if I want to convert 3 degrees Fahrenheit to a temperature difference, I get back this the difference between that temperature and absolute zero. So 462 degrees Fahrenheit, that is the same operation as converting 3 degrees Fahrenheit to Kelvin and then converting that degrees Kelvin into degrees Fahrenheit difference. We get the same value because it's fundamentally the same operation. We're asking for the difference between that and absolute zero. So there is one conversion operation that isn't permitted and that is the conversion from temperature difference units into absolute temperatures. There's not really a physical analog to converting from a difference versus absolute zero to an absolute temperature. And so if you try and convert degrees Celsius to degree degrees Celsius difference to degrees Fahrenheit, you'll get a warning message mentioning that you can't convert between those. If for some reason you imperatively need to do that, the operation is effectively converting that temperature difference into Kelvins, which has a meaningful representation versus absolute zero. And then you can convert that degrees Fahrenheit if you need to. And Kelvin is our common temperature across both absolute and difference units, because it is fundamentally the same on both physical measures. There are also implications to how this distinction between temperature difference and absolute temperatures exist in arithmetic. And so we there are caveats to how addition, subtraction, multiplication, division, etc. Work interacting between the two of these different physical quantities that inhabit the same space of temperature units. So a very common operation would be, I have an ambient temperature, it's 3 degrees Celsius outside, it's 3 degrees Celsius warmer, how warm is it? It's 6 degrees Celsius. So a temperature plus a difference gives us another temperature. And it doesn't matter if you mix the scales, because there's A fundamental fixed value for the temperature and the difference is a change in temperature. You can also freely add temperature differences. One difference plus another difference gives us a value in temperature difference. So 3 degrees Celsius difference plus 3 degrees Celsius difference is the same as 2 times 3 degrees Celsius difference. We get 6 degrees. And we can also convert between scales just fine. For addition of temperatures, there is a change in the way that we're handling temperatures. If you add 3 degrees Celsius to 3 degrees Celsius, you'll notice you are getting 552.2 degrees Kelvin, not 6 degrees Celsius. And that's because when we handle that zero value appropriately at absolute zero, we need to have a uniform operation between adding those temperatures and summing those temperatures if they were converted to Kelvin. So, so we get the same result here, adding 3 degrees Celsius plus 3 degrees Celsius. The point here is just to make sure that this doesn't transfer into multiplication. So we get the same result whether we convert to Kelvin or not, and we return in Kelvin to avoid ambiguity in what the interpretation of the input is. For subtraction, we have the fundamental definition of a temperature difference is the difference between two temperatures. So you subtract one temperature from another, you get a difference regardless of scale. Again, temperature differences can also be subtracted from each other and remain a temperature difference. If a temperature is subtracted, a temperature difference is subtracted from a temperature. It's like adding a temperature difference. We get a temperature back. If you were to take a temperature difference and subtract the temperature, we effectively get the same operation as reversing the difference and then multiplying by minus one. So this is a somewhat counterintuitive result. And when you first look at it, to say 5 degrees Celsius difference, minus 20 degrees Celsius is minus 15 degrees Celsius. But it's fundamentally the same operation here. We'll move through the last couple sections here quickly. I also have this notebook uploaded, so if I'm going a little quickly, feel free to consult the notes there. So this is another case where 2 times 3 is not 6, but 514, because 2 times 3 degrees Fahrenheit is the same operation as 3 degrees Fahrenheit plus plus 3 degrees Fahrenheit. Again, scroll back down and that gives us our. 514 degrees Kelvin. Again, the same operation as first converting to Kelvins and then multiplying by two. There's also a caveat that because of the arbitrary choice of where zero values are, multiplying a absolute temperature by -1 does not necessarily move the sign inside. -3 degrees Celsius is not the same thing as -3 degrees Celsius. So we can take this unevaluated form here. If you look at the input form of that expression, you can see that the minus sign is staying outside the quantity. But the arithmetic framework knows how to handle this. And so if you add that to an absolute temperature, we get back a temperature difference again, as if we were subtracting one temperature from another. This is also true if the if we do multiplication, the sign will remain outside of the quantity even if there is already a minus sign in there. So minus 2 times 3 minus 3 degrees Celsius is not 6 degrees Celsius, it is minus minus 6 degrees Celsius. We'll have time for questions here shortly. But just a quick note on division. That dividing a temperature unit will give you the equivalent of converting that unit into Kelvin beforehand. So one over one degree Celsius is the same as one over convert one degree Celsius to Kelvin. Again, we return Kelvin here by default as a way of avoiding ambiguity in the interpretation. And dividing one temperature by another gives you a ratio just like any other unit division. And the last caveat is that because we distinguish between degrees Fahrenheit as an absolute temperature and degrees Fahrenheit difference as a measure of temperature, rate of change of temperature. If you try and insert a absolute temperature inside a compound unit, we will give a warning message and convert to Kelvin. Because 3 degrees Fahrenheit per meter is almost always going to mean 3 degrees Fahrenheit difference per meter. We are extremely rarely going to run into a scenario where you have an absolute temperature per another physical quantity. And again, this operation is equivalent to taking the original quantity in temperature and dividing it by another unit. Same thing as first converting to Kelvin and then dividing by another unit. I have included in the notebook some additional notes. The Units Tech note has a nice overview of the general units framework, some explanations, and there's a more detailed temperature units tutorial that I've also included in the notes for you to peruse. So I will take a quick look at some of the questions that have come in here. Yeah, so there's a question about is 1 quantity 1 rotations per day the same as quantity 1 rotation divided by quantity 1 days? Either pattern works. It is basically a notion of are you taking one physical quantity and dividing it by another, or do you have a single physical quantity that is rotations per Day generally. So if you use the form where it's one quantity divided by another, it will evaluate to the former form. Either one can be used. It's just whether you want the arithmetic to be performed in real time or if you know that it is going to be performed in advance. Let's see. ITAI asks what we have against Rankine. So if you. Rankine is another zero based temperature scale. It's not commonly used. Neither is Romer. There are also temperature scales like degrees Del C is actually an inverted scale. So the larger the value gets, the colder it is, which is actually historically what Celsius was. Celsius ran from 100 to 0 instead of 0 to 100, but everyone found that too confusing. So the scales were. The inversion was factored out when Celsius wasn't hanging around to complain about it anymore. There's also a question about whether Mathematica throws an error if you do one kelvin minus two kelvin. It does not. So. If you were to subtract. We do allow temperatures colder than absolute zero. We do not have physical limitations there. The framework is flexible, but again the zero value for temperature is based on absolute zero. And so that's our fundamental measuring point. And it's possible to annotate a vector to declare all the elements or a specific quantity unit without having to map. Yes, that is quantity array. For example, I can have range of 1000 meters and I will get basically a fixed array that is able to pack its elements together with a decorated unit without having to generate. I can also say I want range of quantity 1000 meters. This will generate a array or a vector of 1000 quantity expressions, which is what the normalized form of this quantity array would be. But the quantity array is stored more efficiently because it can separate the unit part from the magnitude part. And there's good documentation on quantity array as well. Let's see here how much Wolfram knows about units. A fair amount of non unit functions, which. So in general the symbolic framework for arithmetic understands units. You know, solve understands units. I can say solve x times quantity 3ft equals, let's say speed of the speed of light. And I want to solve for x. I can know that. Okay, it's selecting hertz as a unit here, but we know we need reciprocal seconds. It can do the dimensional analysis to handle that. Numeric functions also generally know how to handle units. So do the. The derivative and integrate frameworks. And the statistics framework has a whole quantity distribution system that handles probabilistic distributions with units attached to them. There are going to be some areas where it isn't heavily optimized and they're going to be some areas at the fringes where we may not have attached units. But in general, when we find areas where units aren't working properly or they aren't working very efficiently, we try and plug things in and work as efficiently as possible. So, you know, the image framework doesn't fundamentally work with quantity under the hood and saying these are all quantity of n pixels or anything like that. But there is an interface for handling pretty much anything that has metadata or anything that needs dimensionality to understand the unit framework and handle conversion where appropriate. Let's see examples of. Let's see. So we can do our. I'm trying to remember what is a built in unit that has a fractional exponent for powers. Say we can check actually. And let's see. Yeah, so here are some examples of electron chart. Yeah, so many of these are going to be kind of esoteric units, but we might say, okay, TS is a unit I'm not super familiar with, but if we say unit convert of 2T, we will get back a value that is mass per seconds to the 5/2 kelvin. So we don't really have a. Simpler common representation for that. I'm not sure what that physical quantity actually associates with, but there are some cases there, or for example, we have real valued exponents. So I won't dig into that too much here. See how much Victor. Okay, that looks like that covered most of the questions that are here. Yes, gravitational constant is a good example. Anton Antonov asks if we should have had units from version one. The anecdote I'll share is that the project that became WolframAlpha largely started as an incubation project to try and properly address units in Wolfram language. And it's a big problem. A lot of our work on the unit framework in Wolfram language has been informed from work done in WolframAlpha. But doing units right, doing a basic version of units is relatively straightforward. Doing them correctly for all cases is very complicated. We introduced units inversion, not the quantity framework in version nine and we've been refining it ever since and making it better and faster. But it's always an ongoing process and there are lots of tendrils once you get to edge cases. It looks like I am slightly over time. So I'm going to go ahead and stop sharing here and hope everyone enjoyed the talk. As I said, my name is Nick Laraveer and I work at Wolfram Research. I'm our lead quantity and units developer. So if you have any questions, feel free to contact us and they will find their way to me. Yeah. With that, I will let everyone migrate on to additional talks.