Skip to main content

Using Redis Lua scripting for complex queues

We've just published a short summary of how we are using Redis Lua scripts to build high-performance queues, which ensure data consistency. It's here, on IPR blog: http://blog.ideaportriga.com/using-redis-lua-scripting-for-complex-queues/

Comments

Popular posts from this blog

Performance of Redis sorted set operations

I was working on a feature recently, which involved several Redis "remove from sorted set" operations. Target production environment sorted sets are expected to be small and most of these calls would be trying to remove items, which do not exist in the sets. Although ZREM operation has documented LON(N) time complexity, ZSCORE has documented constant time complexity. This led me to believe, that Redis might have constant time complecity for ZREM calls when values to be removed do not exist in the set (if ZSCORE "knows" if item is NOT in the set in constant time, ZREM can do the same, right?). Not in this case, apparently. ZSCORE documented constant time complexity is actually misleading (as many cases of asymptotic time complexity documentation for small data sets). Redis stores small sorted sets (up to 128 items by default) as "ziplists", which are essentially linked lists, optimized for memory consumption. Browsing through Redis source code confirms th

Lost promises

I love Promises . I think they make modern JavaScript possible, especially on the server side. But promises are, you know, promises and some of them are literally lost! I would even say that lost promises are, to a certain degree, the buffer overflow of JavaScript. OK, it's not as widespread and it hasn't cost as many billions of dollars, but it still may be as subtle, as difficult to notice and just as devastating. At least I have encountered this issue a few times and it works like that: In the code above we simply forget to add "return" keyword before call to sideEffect3 function. This is totally OK, except when you rely on the fact that the Promise returned from giveMePromise is resolved after "side effect 3" can be observed. In our case, Promise was given, but it was lost. That sideEffect3  function is trying in vain, because it's work will never be used. I think this is just a danger of asynchronous code and such errors can only be detecte

Redis Lua scripts are not really transactions

Redis support of Lua scripts is a great feature. We use it a lot to build fast reliable queues with some very interesting requirements. You need it every time you want to decide your next Redis command, based on the result of a previous command, while guaranteeing that no one else has done anything with this result or anything else has changed in Redis. That is, the whole Redis script is an "atomic" operation. However, I put it in quotes intentionally. My understanding of phrase "atomic operation" is that not only no one else can see it half complete while it is executing (that works so great in Redis). It should also mean, that it should never be left half complete if an error occurs in the middle (or at least, that is my wishful thinking:) ). Yea, exactly, the second point doesn't work in Redis and there is no warning in the official docs. To be more polite (or precise), there is no rollback in Redis (referring to a comment in this SO question -  http://sta